Cribl: The Stream Life

Welcome to Cribl: The Stream Life, a podcast for IT pros trying to take control of their observability data with a no-compromise approach. With each episode, our hosts will cover the latest insights, trends, and emerging technologies to help IT organizations achieve observability in their operations. We’ll also address specific challenges we’ve seen with hundreds of enterprises over the last several years and sketch out the fundamental capabilities required to overcome them.

Join Cribl at RSA Conference 2024! 04/29/2024

Storm Drains and Data Lakes 04/22/2024

Introducing Cribl Lake! 04/17/2024

Engineering at Cribl 04/02/2024

Cribl Announces Partner Award Winners! 03/07/2024

CriblCon 2024 03/06/2024

Cribl for Startups 01/24/2024

How the All-In Comprehensive Design Fits Into the Cribl Stream Reference Architecture 01/12/2024

How the All-In Comprehensive Design Fits Into the Cribl Stream Reference Architecture In this livestream, Ahmed Kira and I provided more details about the Cribl Stream Reference Architecture, which is designed to help observability admins achieve faster and more valuable stream deployment. We explained the guidelines for deploying the comprehensive reference architecture to meet the needs of large customers with diverse, high-volume data flows. Then, we shared different use cases and discussed their pros and cons. provide a way for admins to get 70% of the way towards deploying . The sample environment below is a template for sending data to many destinations while minimizing data egress costs. It incorporates solutions to some of the challenges typical larger organizations might face. MS Azure Worker Group In this sample environment, the leader is up in and managed by Cribl. On the right-hand side, you’ll see an Azure worker group. There are two reasons to consider putting a worker group in a different cloud provider. The first is to be as close to the data you're collecting as possible. By keeping the data close, you can minimize the amount of processing necessary and cut egress costs. With this setup, you’re also reducing the risks of having competing workloads. Failing small is much better than failing big. Additionally, when establishing a security or observability data lake, you don't need to put all that data in the same data lake, S3 bucket, or blob storage. With Cribl, you can have them in different places and still be able to replay against all of that data. We often see customers with Azure and AWS workers using Cribl-to-Cribl connectivity between the two clouds to exchange data. This way, they can avoid building custom code or dealing with the vagaries of exchanging data between clouds. On-Prem General-Purpose Worker Group The next worker group in our sample architecture above is an on-prem, general-purpose worker group. With this worker group, you can combine most of your data sources and have them go to one worker group in your data center. This is especially useful if you have a lot of Splunk universal forwarders, agents, and Filebeat agents — you'll want to send those to a dedicated worker group so you're not competing for different workloads. Another big reason for this approach is segmentation. For example, if you need to separate your PCI or PHI workflow, you can use this setup to break up your data or meet compliance requirements. If you need to upload that data to an Elastic or Splunk cloud, having the Cribl Stream worker group allows you to stage your data, manage it, and get it to those destinations. Syslog Worker Group Another architectural consideration worth looking into is having one Syslog worker group. This allows you to do your commit-and-deploys once instead of one region at a time. A lot of organizations struggle with the contention that high-volume Syslog causes. Adding an agent workload can make the situation worse, so having separate worker groups allows you to scale. The difference between this worker group and others is Syslog groups have load balancers that will send data to the local workers in that data center. In Cribl Stream, there will still be one logical Syslog worker group to manage, reducing administrative burden and the maintenance required. If you take one thing away from reading this post or watching the live stream, please DO NOT send your data to a single Syslog destination port! You'll get the best results by getting as many workers involved as possible — do everything you can to avoid being pinned to a single core. Cribl Cloud Worker Group With Cribl Cloud, you will also get at least one worker group by default that you can allocate to all your AWS data sources — like in the sample architecture. But you can also send all of your cloud, on-prem, and other non-AWS data sources there. Either way, you won't have to manage as much infrastructure. Instead, you can leverage the Cribl Cloud worker group and the Cribl Cloud leader if your use case allows for it. This is especially important for threat surface reduction. Taking data in from multiple SaaS platforms means opening up your perimeter to everything that Cloudflare could produce, which is probably half the entire internet. Cribl Cloud can handle all of those threats and keep you secure. Replay Worker Group The last worker group in this reference architecture that people don't typically consider is the Replay worker group. It’s a great practice to allocate your replays to a separate worker group, where the workload can be spun up and spun down — instead of on your production worker groups where you're processing real-time streaming data. Using your production worker group for replay can suddenly add terabytes of data to your existing live data flows and slow everything down. A minimal-cost, ephemeral replay worker group lets you scale up to meet your needs without interrupting your production workloads. A recent customer took advantage of this by deploying their replay worker group in AWS ECS. As more data gets requested and downloaded, ECS spins up additional instances. The worker group scales larger as more data is retrieved and then scales down if there’s nothing to do. Choice and Control Over All of Your Data When you have multiple worker groups, you don’t have to worry about going to different places to manage them — it can all still be done by one Cribl leader. You can also have multiple data lakes and replay from all of them via one central location within Cribl. This flexibility gives you complete control to make the best choices for you. So, if your security team wants to use Azure for its data lake and your operations team wants to use AWS, it’s no problem. Or, if you want to use one S3 bucket for forensics and another for yearly retention, you have that option available. The best part is that all the data in your data lake is vendor-neutral. You can return that data to Cribl Stream using replay and send it to any tool you want. Check out the full live stream for insights on integrating Cribl Stream into any environment, enabling faster value realization with minimal effort. Our goal is to assist SecOps and Observability data admins in spending less time figuring out how to use Cribl Stream and more time getting value. Don't miss out on this opportunity to enhance your observability administration skills. More Videos in our Cribl Reference Architecture Series /episode/index/show/cribl/id/29198998

How SpyCloud Architected Its Cribl Stream Deployment 11/21/2023

How SpyCloud Architected Its Cribl Stream Deployment In this livestream, I talked to - Manager of Security Operations at , about how he used the to build a scalable deployment. He explained how this approach enabled SpyCloud to grow alongside its evolving needs without requiring significant rework. The reference architecture also facilitated a repeatable data-onboarding process, reducing administrative time and allowing the team to focus on critical security and data analysis tasks. SpyCloud is a cloud-native organization that generates enormous amounts of data — from hosted email and EDR, sales solutions, and the rest of their sprawling cloud architecture. Before implementing , they had too many sources and too little time to figure out how to integrate all of them. Saving Valuable Engineering Time Traditional on-prem environments can have many sources, but they generally come from a single area that makes it possible to capture them with a single set of agents. Because of their sprawling cloud architecture, Ryan and his team didn’t have that luxury. During our conversation, Ryan pointed out that engineers come to work at SpyCloud to work in security, not to become a data butler. They don't necessarily know how to architect large data pipelines — they just pull the data in and go to work on it. To that end, the first problem they solved with Cribl Stream was streamlining the process of bringing sources into their detection analytics platform. Data now flows in natively from a source like AWS instead of via a TA or other inefficient, incomplete method. Flexibility in Scaling Security Architecture SpyCloud can’t afford to have data held up in processing — once all their data comes in, it needs to be processed immediately so their security detections fire in real-time. Cribl’s Reference Architecture played a very important role in onboarding their sources and getting things to operate seamlessly. There are times when Ryan and his team get little to no advance notice of a new product or customer, so there may not be much time to add to their logging pipeline. Without Cribl Stream, planning and execution may take weeks or months. But the right tools and a properly designed architecture allow them to scale up in minutes, if not automatically. Splitting Up Worker Groups Spycloud separates worker groups based on data volume workflow and as a way to mitigate risk. Instead of having one large worker group, they have a separate one on the internet with open ports, so they’re able to fail small and manage their blast radius. It’s good practice to split up your worker groups not only by load, but also by connection type and according to your security needs. When I asked Ryan if he was concerned about the management overhead of having a bunch of worker groups, he compared the experience to his days as a Splunk admin. Setting up different indexer clusters was a nightmare because maintenance efforts only scaled linearly. With worker groups, there’s one interface to manage everything. Ryan can copy settings by cloning a worker group, or add and remove pipelines from different worker groups — all from one interface. He sums it up quite nicely: “The biggest win for us with Cribl Stream is that we can upgrade everything from one single pane of glass. I don't have to go out and plan a 12-hour overnight weekend upgrade of my indexers. I just click upgrade in that worker group, and it happens.” - Ryan Saunders, Manager of Security Operations at SpyCloud Taking Advantage of Cribl Edge Ryan and the team at SpyCloud also have deployed as a log collection agent on all their servers. They have a dozen Edge fleets collecting data that’s sent back to Cribl Stream for processing. Managing fleets in Cribl Edge is just as easy as managing worker groups in Cribl Stream. They have the flexibility to control separate configurations for Windows, Linux, production tests, and other products within the same interface. SpyCloud also uses Cribl Edge to consolidate logging agents within the organization because it’s easier for them to have one agent that multiple teams can control. His team sends the data they need for security to their own tools, and their DevOps teams can extract the operations data they need as well. Everyone can control and manage their data however they see fit, so it's a win for everybody. Best Practices for a Scalable Cribl Stream Deployment Ryan has many years of experience using Cribl’s tools within different organizations and environments, so he has learned some very valuable lessons along the way. His first deployment involved trying to run Kubernetes in a large environment with one giant worker group — so he quickly learned about the importance of splitting them up. You want to be able to do this easily, especially in highly regulated environments. Multinational organizations may not be able to commingle data or send it across national borders. Companies processing healthcare data have strict requirements for handling PII. Even if you don’t fall into either of these categories today, business growth or regulatory requirements might change that, so you’ll need to be able to adjust quickly to split certain data out. Taking advantage of auto-scaling has also proven beneficial for Ryan, and everyone can take advantage of it — just don’t forget to create limits. You want to avoid scaling up until an AWS region explodes, so you don’t wake up one night and find 1000 Kubernetes nodes running because something went sideways. Explaining that bill won’t be much fun the next day. to see more on how SpyCloud uses and to streamline the onboarding process and get more visibility and insights from their business data. You’ll also learn how to use the as a starting point for a scalable deployment so you can reduce administrative time and free up your team to focus on critical security and data analysis tasks. More Videos in our Cribl Reference Architecture Series /episode/index/show/cribl/id/28741163

Modernize Your SIEM Architecture 11/16/2023

Modernize Your SIEM Architecture In this Livestream conversation, I spoke with John Alves from CyberOne Security about the struggles teams face in modernizing a , controlling costs, and extracting optimal value from their systems. We delve into the issues around single system-of-analysis solutions that attempt to solve detection and analytics use cases within the same tool. We explored the strategic limitations of this type of security architecture, presenting alternative options for effectively mixing and matching data platforms. Be sure to watch the full conversation to get on the path toward achieving the optimal combination of data management and cost control capabilities. If your security architecture is centered around a SIEM that houses all your security and operational data, it’s time for an upgrade. Data quantities, cyber attacks, and regulatory requirements are all on the rise, so having a single destination for your data leaves too much room for vulnerabilities. Until recently, buying a SIEM meant deploying its agents, putting all your data into it, and going on your merry way. You were almost 100% confined to that one framework — if you wanted to use UEBA, your vendor or one of their partners provided it. Operating outside your SIEM or bringing in third-party vendors was very limited. Observability Pipelines to the Rescue About five years ago, the concept of an observability pipeline emerged, allowing organizations to funnel their observability and security data through a consistent data plane. The idea of controlling where your data gets stored was born, and vendor-neutral considerations began gaining popularity. Admins can now make copies of events for their SIEM, , solution, or someone else's data lake — easily turning one event into four events that power different parts of their security stack. By moving data into a data lake instead, admins can analyze data and build dashboards for operations teams without bloating their ingest. Teams have more choice and control over their data than ever before, so they can consider their specific needs when building out their infrastructure. The Benefits of a Data Security Lake During our discussion, John mentioned how this flexibility is no longer a wish-list item for his clients, but a necessity. As the industry transitions to cloud infrastructure and cloud-based computing, organizations require vendor-neutral data that supports their scalability efforts. There are a host of benefits you get from modernizing your security architecture. Reduced License Costs Routing data that isn’t needed for security to object storage is one of the best ways to reduce SIEM license costs. Ingest costs go down, and you avoid the upsell for archive data — around a 4- 8x markup — as opposed to using your own object storage or your SIEM cloud platforms archive. You can also store it in a vendor-neutral format, giving you enormous flexibility that you wouldn’t get otherwise. We recently worked with a developer team and their debug logs, routing them to a lower-cost S3 bucket instead of their SIEM. All we had to do was create a rule in to route them to the data lake, and now they’re available to be restored whenever necessary. This is just one example of many where we can set customers up to meet their simultaneous need for availability but lower cost and overhead. Increasing Security While Decreasing Engineering Time When you can reduce your SIEM license costs, you no longer have to choose which data sources you can afford to collect. By removing the constraints for engineers that come from not having the raw data when needed, security teams can focus on security and not just moving data around. No more time spent on tasks like going out to a server to manually zip up and pull in logs. The result? Better detections, analytics, and security. Shared Data Within the Organization Each team has a different use case for the data the organization collects — having different pipelines to transform and send data to different sources is invaluable. Putting firewall, threat, traffic, and systems logs into a single destination is a great way to bloat your ingest. And not all logs from a single data source are security relevant. Routing some of them into a storage account or data lake will not only save on ingestion costs and create less noise for security teams, but you can also give access to relevant logs to your infrastructure, firewall, and other teams. Route your threat logs straight into the SIM, but send traffic and other logs straight into the data lake for your infrastructure network team. Compliance With Retention Requirements Another benefit of keeping raw copies of data is complying with retention requirements. If you're manipulating data before it goes into your SIEM, then you’re not adhering to some necessary standards. Transform events to get what you need for your SIEM, but keep unmanipulated, raw copies in your data lake. Your IR or legal counsel can control forensic copies. Meet Cyber Insurance Requirements As insurance companies get more sophisticated and start hiring engineers as auditors, they’ll dive deeper into your architecture than before. They’ll ensure you have a SIEM in place but also check to see if you’re putting the right data in and using it appropriately. Government auditors will want to see all your data sources and detections. They’ll be ready to write findings if you’re not following best practices. The prevalence of bad data or an overwhelming amount of data leads to various issues with detection, and drives costs higher and higher. It is extremely common to witness a year-over-year cost increase of up to 35%, which is clearly unsustainable. to hear John and I talk about alternative options for your SIEM platform, so you can be empowered to re-architect your data strategy. With the right strategies, SIEM platform challenges can be overcome, and we’re here to help as you embark on this transformative journey. /episode/index/show/cribl/id/28390316

Solving Data Challenges with Adam Hogan from CrowdStrike 10/27/2023

Hackers Aren’t Hacking Into Your Network -- They’re Just Logging In 10/26/2023

Unpacking the Hype: Navigating the Complexities of Advanced Data Analytics in Cybersecurity 10/16/2023

The Gartner Hype Cycle for Observability and Monitoring 09/13/2023

Reference Architecture Series: Scaling Syslog 09/11/2023

What are Telemetry Pipelines? 08/16/2023

Cribl and Exabeam Aim to Accelerate Technology Adoption for Customers 08/10/2023

Cribl's Enhanced Authorization Support 07/25/2023

What's New With Cribl Stream, Edge, and Search? - Summer Launch 07/19/2023

Building a Distributed Security Team 07/13/2023

The Top 4 Trends Defining Observability in 2023 06/14/2023

The Evils of Data Debt 06/08/2023

CriblCon - Criblers: Assemble! 05/25/2023

Industry Experts Discuss Cybersecurity Trends and a New Fund to Shape the Future 05/04/2023

Giving the Goats A Day Off 04/28/2023

It's Time to Assess the Potential Dangers of an Increasingly Connected World 04/21/2023

All About Cribl's Partner Program 04/11/2023

The Critical Role of Data in Cybersecurity: Why Incomplete Data Weakens Your Overall Program 04/05/2023

The Critical Role of Data in Cybersecurity: Why Incomplete Data Weakens Your Overall Program In this live stream, and discuss a foundational issue with many security programs — having the right data to detect issues and make fast decisions. Data drives every facet of security, so bad or incomplete data weakens your overall program. Watch the video or continue reading below to learn about these issues and the strategies we use to solve security's data problem. As the amount of data, tools, systems, and clouds continue to increase, the threat to enterprises' security posture has risen as well. It simply doesn’t matter what kind of SIEM you have anymore — even if it’s as good as Splunk or its alternatives. If you don’t have the right data, you’ll run into problems. The Problem with Dropping Data Sources Due to Budget Constraints Budgets can no longer keep up with the amount of data that needs to be processed, so organizations are forced to get by without collecting and analyzing everything they should. As a result, security teams are forced to turn off data sources that could provide them valuable insights into credible threats. One client that Brenden and the team at worked with got a firsthand look at the effects this has during a pen test they performed. They tested some common detections and were surprised to find that their red team engineer was able to completely compromise the domain and gain full control — simply because they had turned off all audit events on Kerberos. Situations like this are much too common and are just the tip of the iceberg —which is why it’s so critical to have visibility into all areas of your network. You also need someone who knows all the different attack vectors so they can help you set up your infrastructure to avoid them. Poorly Formatted but Crucial Data Sources Eat Up Licensing Costs Data sources like Powershell, Sysmon, and Windows DNS debug logs are generally more difficult to work with. In the past, you’d have to rely on the heavy forwarder on the Splunk side or a ton of manual fine-tuning of things on the source side to handle the flood of data coming in from all these different systems and formats. This is where a tool like can help — you can turn on a data source, send it to Stream, and then route to null by default. Then you can pull out specific streams and send them to your other tools as necessary. Other data won’t need to be processed but will need to be kept for regulatory compliance issues, so you can keep it offline in raw, unmodified form in a data lake or send it to an object storage like an S3 bucket for as long as you need. Then if you need to recall it to investigate a data breach, you can to ingest it back through to whatever source you want without having to use your license or processing power. You can also use Cribl Stream to take advantage of EDR data. We see a lot of companies make enormous investments in EDR tools that also produce very accurate data, especially around assets — but then they don't take that data and put it into their SIEM because it's just too expensive. With Stream, you can take the majority of that EDR data and route it to a data lake, and then get value from the other 10-15% by routing it to your SIEM in the exact format you need it. Data Volume Management Strategies to Get the Best Results for Security To get the most value out of your data for security, you need to know what regulatory compliance you have to meet — what type of logs do you have to retain, and for how long? It also helps to have a good understanding of all the tools you have, what systems are in place, and what the limits are on your ingestion licenses. From there, securing your perimeter is the best place to start. You want your authentication sources, MFA sources, and VPN set up first, and then you can start bringing in all your security tools. The Mitre Attack framework is incredibly helpful to figure out what vertical you’re in and see the common threat actors or attacks right you might encounter so you can decide which sources and services you’ll need visibility from. Having had a long career in IT, I became used to constraints and compromise — which is why I was caught off guard when I first saw Cribl Stream back before I joined the company. Not having to make concessions on which data to pull in, where I could send it, what format it was in, or what my vendor would support was unexpected, to say the least. This choice and control is giving security teams the ability to have faster detections and even better responses to cyber threats. Be sure to , and connect with us in our if you have any questions or want to continue the discussion! /episode/index/show/cribl/id/26307366

Exploring Platform Engineering: Impacts on Observability and Security 04/03/2023

The Cribl Curious Relaunch! 03/29/2023

Cribl: The Stream Life

TOPICS