Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Tech Life

Towards More Inclusive Engineering in Artificial Intelligence

Any scalable AI within an organization is bound to become more inclusive, as businesses need to think carefully about their product and their needs. Ultimately, it’s the company leaders’ responsibility to make that call.
May 24th, 2021 1:00pm by
Featued image for: Towards More Inclusive Engineering in Artificial Intelligence
Feature image via Pixabay.

Conor Jensen
Conor Jensen is an experienced Data Science executive with over 15 years working in the analytics space. He is the Vice President of AI Consulting and Data Science for the Americas at Dataiku, helping customers with their biggest and most ambitious AI and Data Science projects. He has worked at a couple of Data Science tech companies for the last few years after taking his lumps building out data science teams and platforms at multiple large enterprises. In prior lives, Conor was a Weather Forecaster in the Air Force, a Starbucks Store Manager, an Emergency Management Technician in the Air National Guard, and taught archery and rock climbing. Somewhere along the way he also earned a Bachelor’s in Mathematics from UIC.

When it comes to the history of data teams and data-specific roles, we’re still very much in the wild west — that is, brand new territories where everything is kind of rough around the edges and (mostly) lawless. Sure, some best practices have come into focus in recent years, but for the most part, there isn’t one proven right way to do things; the fact that data professionals’ job titles (and the roles themselves) vary widely is further evidence of this.

One such fork in the road for the future of data teams, data roles, and even for artificial intelligence (AI) as a whole is the question of whether AI should be inclusive (that is, encompassing all different types of people across roles working together toward a common goal) or exclusive (siloed to specific and specialized teams to get the job done more precisely and efficiently).

In which direction AI veers has the power to change the core organizational structure within companies, not to mention individuals’ career paths. So what will it be — inclusive, or exclusive?

It’s hard not to argue that the roots of AI are exclusive. After all, it’s the product of data science, which means all kinds of fancy and exclusive education (today this is changing, but at first this was certainly the case ) — Ph.D.s, statisticians, mathematicians, etc.

The very role of data scientist continues to be put on a pedestal as the so-called sexiest job of the 21st century. So much so that when the term “citizen data scientist” arose a few years ago, it’s not difficult to see why it was looked down upon seeing the exclusive image of the domain.

While I don’t necessarily agree with the term or concept of “citizen data scientist” (that’s another debate for another time), the larger idea that AI could — or should — move from being something completely exclusive to inclusive is an interesting and essential one.

Inclusive AI: What and Why

Today, inclusive AI means several things:

  1. Collaboration on AI projects between different people with different profiles, strengths, and educational backgrounds; by nature, this also usually means different departments working together to achieve a common goal.
  2. But more broadly than simple collaboration for one particular project, it also translates into the wider infusion of AI processes throughout an organization — a complete transformation in the way of working.
  3. Lastly, but perhaps most importantly today, inclusive AI also has started to take on a slightly different meaning that is outside of the way businesses and companies work internally. It deals with issues like bias, responsibility, interpretability, and fairness — areas where we are also arguably in a wild west situation.

Overall, inclusive AI encompasses the idea that the more people are involved in AI processes, the better the outcome (both internally and externally) because of a diversification of skills, points of view, or use cases. Practically within a business, it means not restricting the use of data or AI systems to specific teams or roles, but rather equipping and empowering everyone at the company to make day-to-day decisions, as well as larger process changes, with data at the core.

But is it the be-all and end-all? Is it the direction that companies should take?

Exclusive AI: The Alternative

For lack of a better opposite word (non-inclusive?), exclusive AI would be the alternative to the above presented scenario. It’s an interesting word because it can have both negative connotations (restriction, irreconcilable) as well as positive (premier, dedicated, complete). Similarly, it could be both negative or positive in the enterprise, depending on how it’s implemented.

A future of exclusive AI probably wouldn’t look that much like its current exclusivity, and that’s probably a good thing. Keeping AI literally in the hands of just a few data scientists would not be good news for scalability. However, it would be much more localized, allowing those truly specialized in AI systems to move quickly and not be encumbered by those with less experience in the space.

On the positive side, that would mean much faster advances in Enterprise AI, to be sure. But at what expense? Perhaps with exclusive AI, systems won’t be as tailored to business needs, or worse, they will be PR disasters (if they turn out to be biased or produce results the company can’t explain).

In the end, exclusive AI might be well adapted for companies needing to quickly deliver a new product and disrupt their market through a technological advantage. But any scalable AI within an organization is bound to become more inclusive, as businesses need to think carefully about their product and their needs. Ultimately, it’s the company leaders’ responsibility to make that call. After all, determining an exclusive or inclusive approach to AI engineering is likely to shape your company’s culture for years to come.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Dataiku.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.