The statements below first appeared in the r/BusinessIntelligence subreddit.
I have been working as a BI Developer/Consultant for the past 5 years after graduating from University. Many people are thinking about a career in this field. I thought I would offer my perspective of the problems I have faced and what led to my decision to move away from BI. Would love to hear any opinions/advice from others.
The first point I want to raise is that things have changed A LOT in BI/Data jobs over the past 5 years and not for the better. The job does not carry the same level of respect or 'perceived' value in an organization. Before you all murder me, let me explain. Data has more value than ever, I agree. However, the people who extract, clean, combine and deliver this data have much lower value. I am not sure why this has developed.
Job title of BI sounds fancy to most people. Salary ramp-up to mid level ($80k) on par or better than other IT/Business fields. (BI does cap out much earlier than other fields).
Easy to get into a low workload job as a Excel/PowerBI/Tableau data cruncher with a mid-level salary. Progress after that is very hard unless you make shifts to other areas.
Work that nobody wants to do gets dumped into the BI department. Its role is less well defined and it's easy to sneak the mistakes of others into "the data department." There's no systematic way of managing the quality of what arrives in. Once we've taken custody of it and a few days have passed, it's our problem. As if somehow 7,000 emails got turned into NULL in the 2 days since you sent me your file.
I once worked with a client that ran a yearly survey to gather data. They produced a report of top 100 companies and industry trends. Nobody in the client's company wanted to sift through over 10,000 survey responses. Nobody wanted to clean data, extract insights from survey responses. So they just sent it.
This entire workload fell to us. the external consulting company, even with our $150-per-hour bill rate. It took us weeks of work and the company paid out quite a bit. Of course, remember I did not see $150-per-hour for this work, I just received my salary, which was in the $60k range. So who benefited and who overpaid?
Another example, this time from a large enterprise. Daily data loads extract data from [HR, finance, payroll, etc.] systems. New employees are sometimes set up with different/wrong values in different systems. This causes major issues in reporting/BI tools. Senior Management was quick to blame BI. They didn't consider the inefficient processes, or mistakes at the operational level that led to this. The HR/Finance analysts don't care about these issues. It got so bad, eventually setting up new employees in the HR system fell to BI analysts. They main reason was that they cared the most about the data.
The end users look at the data once a month if at all. The weekly emailed static reports often go unread. Instead the end users revert back to the prior solution where data is sourced by BI analysts manually. Guess what the reason was? End users find it boring to have to use cubes to browse data or PowerBI/Tableau to manipulate data. They prefer to file a request with the BI team and let them do that work, or have analysts send them a weekly email. Or simply sit in a meeting where someone else tells them what's going on.
Salary cap to what BI developers can earn. I find that as a BI developer, my salary peaks at around 80% of what other types of developers earn at upper levels. Market rate for me is 90-100K (USD) in house and 100-120K (USD) consulting.
This is made worse by the number of senior SQL server/DBA/BI consultants (+20 experience) in the market. You don't need more than 3/4 years experience with a BI toolset to get the job done properly. Yet I have been on many projects where clients have asked for someone with 12+ years experience. They're later surprised when they learn someone with 4 years experience did the projects.
Job tied to a tool/industry. I was never sure why this matters so much. The ability to learn a new tool to get the job done is under-appreciated. I have worked in finance/retail/media and government BI. But I have been told I am not skilled enough to work in x industry or with y tool that varies slightly. Add to this jobs where I see people with masters or PhD level education doing BI Analyst work. People are on-average under-utilized, in my opinion.
BI testing. The most boring, manual, but most necessary part of any BI project.
Testing SQL business logic is painful because of the lack of automated testing solutions used across companies .
Testing with popular tools (PowerBI, Tableau) is nearly always manual . (Good luck testing complex finance dashboards with complex DAX business logic.)
Source system testing is non-existent. (What happens if you change the time zone in a source finance application. Does all the data for the user we extract change at a DB level as well?)
ETL testing (good luck testing 100+ SSIS packages).
Data Warehouse testing: all too often, complex business logic is piled on top of existing logic due to source system upgrades. cube/dashboard testing. No automated solutions exist. Mainly manual.
It's rare to find business users who will agree to do testing properly. I have seen business users resign from jobs rather than sit and test large amounts of data manually.
While a career in BI is still very attractive to knowledge workers, I wanted to share the pitfalls. I hope my experience helps others. The space still has some maturing to do. If you get with the right organization, it can still be a great career. If they let you use the right data analysis tools, it can still be a win. The key is being able to quickly understand the environment and make quick decisions.
As an employee, you should be watchful for this, but you do have some choices . As a consultant - as I was/am - you'll often get dragged into some of the worst environments to help fix things.
Expect that.
One can easily find themselves stuck cleaning data in Google Sheets for most of each day. It's important to recognize the signs and signals of a good BI vs. a bad BI environment. My advice: look for places where business users are actively involved in BI projects. Companies that invest in their data, and in advanced AI tools. Places where they actually care about the outcome and respect the work you do. Because it's important. You're important.
Good luck out there.
The statements above first appeared in the r/BusinessIntelligence subreddit.
