Top 10 Things We Are Hearing From Our Customers about Data and Search

New CHAOSSEARCH customers and prospects are coming up to speed every week. What we are hearing from them validates our product strategy and philosophy that you should be able to store everything, and ask anything in a cost-effective manner. As our CEO Les Yetton articulated after we closed our Series A this past fall, we have a vision of building a data platform on top of AWS S3 to completely disrupt the log and event management space. What we are hearing from our early customers is that our vision and product are in precise alignment.

So here we go, this is what we’re hearing…

1. Companies want to store more data. Data is exploding at an alarming rate due to growth in Mobile computing, IoT, connected commercial and consumer devices, edge applications, and cloud computing. Companies want to differentiate themselves by innovating and providing new value from data. Companies who are employing new technologies, machine learning, and artificial intelligence are gaining a competitive advantage over those who are not.
2. Storing more data should not be cost prohibitive or be a disproportionate percentage of IT spend. Many of our prospects and customers are alarmed when they realize a surprisingly large percentage of their budget goes into their search infrastructure (Elasticsearch/ELK Stack or other) in hard and soft costs. The desire to store more with longer retention periods is exacerbating real pain for Elasticsearch customers as they face budgetary constraints and in some cases compliance requirements.
3. Everyone sees Amazon S3 as the perfect place to store data at scale. Cost, security, resiliency, and global points of presence make it the standard place to store data at scale. S3 is bedrock for the cloud and sets the bar for all cloud vendors and vendors of on-premise object storage. As Pete Cheslock pointed out, S3 was a major focus of re:Invent 2018 and continues to offer more value as a platform for customers. Everyone loves the economics and functionality of S3, but they are often unable to get value directly in S3. Customers would love the ability to be able to quickly harness value from the data they store in S3.
4. Many organizations are not happy with the complexity and cost of having to manage and maintain Elasticsearch. This is true for both on-prem and in-cloud deployments and even the cloud services offering Elasticsearch solutions. Whether running their own ELK Stack or a managed service in the cloud, customers do not want to worry about mapping out what fields are indexed, cluster sizes, sharding, provisioning, patching, security, and the time that it takes to administer a database. As an example, the ELK (free) version lacks features such as security, alerting, and cluster monitoring and is extremely difficult to scale and maintain. The free version of ELK uses Elasticsearch for indexing, Logstash and Beats for data collection, and Kibana for visualization, but does not include features such as security (node to node encryption, firewall), cluster health monitoring, and alerting. Customers have to pay support for every single node in their ELK cluster to use those features. Logstash, the collector component of the ELK cluster, requires customers to write parsers before logs are ingested and indexed on an ELK cluster.
5. Elasticsearch has some inherent limitations. Elasticsearch scales well but with complexity that sometimes comes with the cost of stability, and scaling in AWS comes with additional EC2 and EBS costs. Additionally, Elasticsearch does not have the ability to do relational queries in a way that people typically do in databases (classical SQL joins). Customers would love to be able to do relational queries, aka joins, against data stored in Elasticsearch.
6. Customers want to be able to analyze and run text search on log data easily and cost-effectively from the products and platforms they use. These products and platforms span the gamut from security products from IBM, Cisco, Rapid7, Akamai, and Cloudflare, to Salesforce.com and Martech platforms, to general IT software. Many great SaaS and on-premise products provide great functionality but often overlook how to search and analyze log and event or other data generated by their products.
7. Customers want APIs for programmatic access to data to integrate with other systems and to build applications on top of. In addition to visualizing data in Kibana or in an interface, many organizations have custom integrations that rely on APIs to connect systems.
8. Customers want to know their data is secure, that they are in control of the keys to their data, and they can control who has access to it and can audit and report on when it has been accessed.
9. SaaS vendors have both internal and external needs for search and analytics with data from their platforms as well as a desire to provide more analytic and search capabilities to their customers. Software as service and mobile technologies have created a platform economy where a service can be created and consumers will use that service to obtain goods and services anywhere, anytime.
10. Customers don’t want to be locked into one vendor, whether it’s a cloud provider, SaaS vendor, or Elasticsearch. Customers want to run products in AWS, GCP, Azure, and in private clouds behind their own firewall. It should be easy to get data in and out, and not be a hostage to any one product or platform.

Conclusion

The advent and proliferation of 5G will only accelerate the growth of data in our global economy of devices and interconnected platforms. Whether you are running Elasticsearch, Logz.io, Splunk, Sumo Logic, or considering of any of these solutions, you will be confronted with many of the challenges that our customers are overcoming by harnessing and unlocking the power of data stored in Amazon S3 with CHAOSSEARCH. Overcome these challenges and become a company who is employing new technology and gaining a competitive advantage with the CHAOSSEARCH platform.