loader from loading.io

A Kafka Introduction

Voice of the DBA

Release Date: 08/27/2024

Technology and Privacy show art Technology and Privacy

Voice of the DBA

There can be a big divide among tech professionals on how they view data privacy. Some don't worry too much, and some are very upset about the lack of data privacy and poor data handling practices from many organizations. Most people are probably in the middLe. I do know that every time I post about a Tesla, there are many people who mention the incredible amount of data Tesla collects, and how that makes them nervous. Tesla discloses that they collect, and I get copies of my data with my own logger. I find the data interesting and I glance at it over time to look for trends. I'm still happy...

info_outline
A Simple Solution show art A Simple Solution

Voice of the DBA

In general, I like simple solutions to problems (and ). They can take work, but a simple solution means it's easy to explain to others, easy for others to refactor/modify, and we limit the amount of effort required to ensure others can use the solution. Sometimes I see very clever engineering solutions, but they become cumbersome to implement and maintain, or worse, no one other than the author understands how they work. In any size organization, that's a piece of technical debt that can cause problems over time. Read the rest of

info_outline
Casino Night is Back show art Casino Night is Back

Voice of the DBA

We are bringing back the SQL Server Central Opening Night party at the PASS Data Community Summit this year. This was for many years and I'm looking forward to the conference even more. The party will take place Tuesday night, after the Welcome reception. from 8pm-10pm, once again catered as a casino themed party. I'm using the same company we've used in the past, and we'll have a cash bar and lots of prizes. I've already started to make a shopping list of fun things to give out as random prizes as you enjoy the games. In the spirit of competition, we will save a prize or two for the top chip...

info_outline
The Invisible Disk show art The Invisible Disk

Voice of the DBA

I saw a post recently about , where a software consultant had written down his predictions for the future. In this case, he writes about a computer shrinking smaller than today, essentially to one chip, with docking stations wherever you need them. We're already at the place where many of us have an extremely powerful computer in our pockets in the form of a mobile phone. I'm still amazed that I bought a phone recently with half a terabyte of storage. A few years ago I gave a keynote where I looked at the changes in disk storage, starting with an . The capacity was 5MB. I remember working with...

info_outline
Container Development Work show art Container Development Work

Voice of the DBA

On my new laptop, I only use containers as database servers. I made the decision not to install SQL Server or PostgreSQL and instead work on containers only. I've written lightly about this, but I set up docker-compose files to load different instances of SQL Server and PostgreSQL (and others) and batch files to start and stop them. I've also set dedicated places on my disk where I can drop backup files and access them from the host. It's 2024. I moved to containers on my laptop exclusively for databases for the first time this year. This is despite the fact that I like containers, am...

info_outline
Fifty Percent show art Fifty Percent

Voice of the DBA

Most of us will have more than one job in our career. In fact many of us will likely find a new job in the next five years. I hope I'm not in that group, but I recognize that it's a possibility. We never know when our situation will change, or our employer's situation will change. That is one reason I recommend you keep your resume up to date and continue to work on improving your skills. I saw recently from Brent Ozar, in which someone had asked him if they should apply for a job even though they didn't meet all of the requirements or know all of the desired technologies. Brent recommended...

info_outline
Trying New Technology show art Trying New Technology

Voice of the DBA

I had someone ask me about recently. Would I think that's a good choice for a database. I don't really know. From their blog and some online research, maybe, but it's also a minority player in a niche space. I had a chat recently with someone that had implemented , a graph database. Why that and not Neo4J I asked them? Someone at the company had tried it and recommended it. Not a bad reason, as I think experience with tech is important, but it's not the only thing. Read the rest of

info_outline
A Kafka Introduction show art A Kafka Introduction

Voice of the DBA

I've heard of Kafka before. I know it's an Apache project and you can download or read more at . I knew it was a way of moving data around, some sort of ETL tool useful for moving things around. More like a message and queueing system, which is a tool that seems like a great idea, but one that everyone struggles to work with. And one that seemed complex. The overview is that Kafka is "a distributed system consisting of servers and clients that communicate via a high-performance . It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud...

info_outline
Your Computer Science Education show art Your Computer Science Education

Voice of the DBA

I know a lot of people in this business do not have computer science degrees. While some do, I suspect it's a minority. I'm certainly curious, so if you want to share your education experience in a comment, tell me if you have a degree and what the focus was, as well as answer a few other questions. At recently, professors and from talked about their computer science curriculum for growing the next generation of professionals. Along the way, they also asked the audience these questions: Read the rest of

info_outline
A Lack of Architecture and Planning show art A Lack of Architecture and Planning

Voice of the DBA

A few weeks ago, I was sitting in the audience, waiting for my turn to speak at Just before me, Xe Iaso delivered a funny and thought-provoking talk on . It was very well done and had me feeling nervous about following that session. The talk is a bit of a satirical look at an interview Xe had for a company that tried to get them to derive an architecture for a large distributed system. It was interesting to hear Xe note that often we have architecture diagrams of what we'd like to have, but never an explanation of how we implement a large system, especially one that has to grow as our...

info_outline
 
More Episodes

I've heard of Kafka before. I know it's an Apache project and you can download or read more at https://kafka.apache.org/. I knew it was a way of moving data around, some sort of ETL tool useful for moving things around. More like a message and queueing system, which is a tool that seems like a great idea, but one that everyone struggles to work with.

And one that seemed complex. The overview is that Kafka is "a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments."

Read the rest of A Kafka Introduction