Wondering whether your struggles with the data lake, cloud data platform, or analytics at large are typical? Are you ahead or behind the curve?
ChaosSearch recently commissioned a survey to understand the advantages and setbacks organizations face today in these areas, and we’re excited to share a sneak peek of the results. To uncover more detailed findings from our research, sign up to receive the full report once it’s available here.
Our Cloud Data and Analytics Survey was conducted by Researchscape. They surveyed 1,020 U.S. IT professionals, working in firms with 500+ employees, from September 28th to October 19th, 2021.
The objective of the survey was to discover how organizations are analyzing and activating data to source valuable insights that inform business decisions. The survey dug into these main topics:
- Top challenges organizations are facing with regard to data retention, usage, and management
- Tools and platforms they’re investing in to overcome those challenges
- How much data they are -- and should be -- storing and accessing as they look ahead to 2022 and beyond
Data Lakes Are on the Rise
According to our research survey, most IT professionals see high potential in data lakes. Sixty-nine percent of survey participants responded that their organizations have implemented a data lake, and 23% of respondents have not implemented one yet but have plans to.
Results indicate that a whopping 92% see data lakes as the right solution for centralizing data and analytics going into the future. Furthermore, 87% of participants who already use a data lake shared that it has improved their organizations’ decision-making ability.
The table below reveals the top benefits organizations are recognizing from data lakes:
Read: Data Lakes Are Gaining Maturity, According to 2021 Gartner Hype Cycle™ for Data Management
Yet, Many Data Lakes Struggle
The Cloud Data and Analytics Survey participants also shared their top challenges with cloud data platforms.
Thirty percent of respondents said their biggest analytics challenge is that end-user consumption and visualization tools aren’t directly connected to the data lake. This leads to issues with data duplication and movement.
A common concern: How much time does it take to activate and derive value from a data lake? We learned that IT talent is being wasted because specialists are spending almost as much time preparing the data (6.6 hours per week) as they are analyzing the data (7.2 hours per week). This indicates that data lakes are causing challenges that limit insights and time to value.
Read: Data Lake Challenges: Or, Why Your Data Lake Isn’t Working Out
Overcoming Analytics Challenges
To achieve their analytics goals and overcome obstacles, organizations are looking to technology for help. In fact, 59% of respondents selected “technology” as their top investment choice -- over time, budget, or people -- to best solve existing analytics challenges. (By contrast, no more than 13% chose time, budget, or people, respectively.)
Furthermore, some data lakes are speeding up data analysis. The data shows that of those survey participants with data lakes, 38% respond to data requests within an hour, whereas only 24% of the respondents without data lakes can respond within an hour.
The Log Data Retention Reality
Are organizations utilizing or benefitting from “unlimited” data retention promised by the cloud? To find out, we asked participants how long they retain their data. We also asked to what extent they agreed with the statement: “My department is using all of the data at its disposal to make informed business decisions.”
Eighty-seven percent of respondents agreed to some extent with the statement that they’re using all data at their disposal. But, interestingly, those who agreed are retaining less log data (only 1-6 months) than those who disagreed (7-12 months). As data volumes and retention windows grow, it apparently becomes increasingly difficult to wrangle and analyze that data at scale.
See results below for the average amount of time participants’ companies retain log data:
The results reveal varied opinions about data retention. One certainty is that volumes of log data will continue to increase. Alongside this growth will come a higher demand for unlimited data retention.
READ: Data Retention Policy Guide
Data Security Starts with Retention
Of course, the security of data is essential for the success of an organization. Unfortunately, many organizations are not storing enough log data for it to be useful in analyzing and preventing a cyber security breach.
We discovered that 47% of respondents who retain less than 7 months of log data have experienced a breach in the last year. Comparatively, only 24% of respondents who retain 7+ months of log data experienced a breach in the past year.
Ongoing Cloud Data Deliberations
Gathering cloud data insights from IT professionals provides vital information to continue to match the needs of organizations. Understanding and acknowledging challenges individuals face is the first step in overcoming them.
“Relying on manual data analysis methodologies is going to be one of the biggest hindrances to an organization’s ability to succeed in 2022... Our research supports that by illustrating the challenges many IT teams are facing—and the limitations they experience with tools that aren’t able to scale with the amount of data being produced today. Without an easy, cost-effective, and reliable way to access and analyze all of the data at your disposal, your business is at risk.”
- Thomas Hazel, Founder and CTO, ChaosSearch
Additional Resources
Read the Blog: 2021 Benchmark Report | Log Management and Analytics
Watch the Webinar: Log Analytics for ITOps, DevOps & CloudOps
Check Out the Brief: How a Cloud Data Platform Scales Log Analytics and Fulfills the Data Lake Promise
