The Knowledge Engineering workforce is accountable for Slack’s knowledge lake, analytics dashboards, and different knowledge providers. The workforce’s mission is to empower customers to leverage knowledge to make selections shortly, precisely, and simply. Slack’s knowledge lake grew in dimension from sub-petabyte to over 100 petabytes lately and it now spans hundreds of thousands of tables. Because the complexity of managing this knowledge grew, so did a various workforce of Slack engineers devoted to supporting the ecosystem.
We have now robust feminine illustration amongst engineers in Knowledge at Slack. Our Knowledge Engineering tradition celebrates range in views and experiences. As knowledge complexity intensifies, having a mosaic of inventive drawback solvers from totally different backgrounds is the important thing to navigating intricate challenges with agility and perception.
Let’s dive into the private tales of girls who’re redefining Slack’s knowledge panorama.
- Take heed to Jessica, a current rent, as she navigates the complexities of Pinot and Tableau know-how.
- Observe Nilanjana, Ramya, Shrushti, and Nathalie, seasoned engineers, as they lead large-scale engineering tasks utilizing applied sciences like Spark, Merlin, DataHub, and Secor.
- Lastly, hear from Suzanna, Lakshmi, and Beate, feminine engineering leaders, as they make clear the panorama of progress and alternatives for girls in knowledge engineering.
Hello, I’m Jessica Stewart, and I’m a Senior Software program Engineer on the Knowledge Orchestration workforce. I joined in Could 2023.
My workforce oversees the inner implementations of knowledge workflows utilizing Apache Airflow and Apache Pinot. Working inside our main cluster, which shops terabytes of knowledge, we preserve a system that boasts sub-second question latency and an almost 99.95% question success fee service-level settlement (SLA). This datastore serves us internally by supporting instruments for workers and externally because the spine for Slack’s user-facing analytics dashboards.
A present technical problem we face revolves round migrating from a digital machine setup to a cloud-native Kubernetes infrastructure. Digital machines imply increased infrastructure prices and upkeep overhead, so we have been keen to extend effectivity with a containerized set-up. We have now an inside Slack Kubernetes platform that doesn’t absolutely help Helm charts or Kubernetes providers, so we needed to make some customizations to the open-source resolution. So as to preserve efficiency necessities, we configured a buyer networking set-up that integrates Pinot with our inside Kubernetes platform. Moreover, working with Pinot permits us to delve into each software program and infrastructure layers. We’ve developed customized Python tooling that wraps round Pinot to standardize operations, and we’ve streamlined knowledge ingestion by Airflow pipelines. On the infrastructure aspect, we’ve automated deployments and upkeep duties utilizing Ansible, Kubernetes, and Jenkins.
When interviewing with Slack, my potential co-workers have been supportive and sort, and this continues to be the case as we work on tasks, deal with incidents, and plan future work. I’ve the chance to personal initiatives, work cross-functionally and broaden my ability set — all whereas working with co-workers I belief and luxuriate in.
Hello, I’m Nilanjana Mukherjee, and I’m a Employees Software program Engineer on the Metrics Foundations workforce. I joined in October 2021.
My workforce is accountable for producing actionable datasets and metrics for data-driven decision-making. We additionally preserve excessive knowledge accuracy and desired touchdown time SLAs.
Becoming a member of the Knowledge Engineering workforce two years in the past has been an unbelievable expertise with alternatives and challenges. Amidst my involvement in lots of knowledge modeling tasks, I led a pivotal strategic initiative: a migration of workloads from Hive to Spark 3. Being one of many earliest adopters of Spark at Slack made it troublesome however rewarding. I grew to become the Spark material knowledgeable to over 40 groups throughout Slack.
The journey was difficult however considerably contributed to my private {and professional} growth. I owe a lot of my success to the sources and unwavering help from my managers and workforce members. Not too long ago, I used to be thrilled to be promoted from Senior Engineer to Employees Engineer — a recognition of my progress, and a testomony to the alternatives for development at Slack in knowledge engineering.
Hello, I’m Ramya Sundaresan, and I’m a Senior Software program Engineer on the Metrics Foundations workforce. I joined in Could 2022.
I’ve had the chance to collaborate with two knowledge groups throughout my tenure at Slack: the Knowledge Ingestion workforce and the Metrics Foundations workforce. On the Knowledge Ingestion workforce, we have been entrusted with the essential process of accumulating, processing, and ingesting knowledge into our knowledge warehouse from quite a lot of sources. This included extracting knowledge from Slack’s Vitess software database, managing custom-made log pipelines, and integrating knowledge retrieval pipelines from Google Sheets. Throughout the Metrics Foundations workforce, I discovered widespread floor with Nilanjana as we centered on related areas of curiosity and experience.
My contributions to Slack’s knowledge engineering ecosystem embody implementing Apache Iceberg inside Kafka Join clusters, orchestrating the migration of Airflow to Kubernetes from AWS EC2 (Amazon Net Providers Elastic Compute Cloud) cases, and migrating jobs to Spark3 on AWS EMR6 (Elastic MapReduce) clusters. We closely leverage AWS. Transitioning from Spark 2 on AWS EMR 5 to Spark 3 upgrades on AWS EMR 6 was a strategic endeavor. It was according to a company-wide goal to determine a unified tech stack and diminish reliance on legacy programs like Hive and MapReduce. I led a number of groups by automated workflows and complete documentation leading to a assured and profitable migration. We achieved our migration objective in beneath a 12 months utilizing parallel construct pipelines and rigorous testing. This unlocked higher efficiency and fortified our programs with improved compatibility, interoperability, and long-term help.
My journey as a software program engineer at Slack has given me alternatives to steer, innovate, and contribute meaningfully to my workforce’s targets. Our on-call rotation and incident administration prioritizes workforce member well-being with group and automation. My work provides me loads of satisfaction, however I additionally have to acknowledge a pleasant fixed in my life — my toddler son, who usually contains the colourful Slack emblem every time he doodles, and jogs my memory of the fun past code.
Hello, I’m Shrushti Patel, and I’m a Senior Software program Engineer on the Knowledge Infrastructure workforce. I joined in August 2020.
My workforce owns all of the infrastructure and providers required to ship dependable and well timed knowledge. Being a part of this workforce, I’ve labored on providers like AWS EMR, Airflow, Trino, Secor, and Ranger to handle infrastructure.
Once I joined, Slack had an information ingestion setup reliant on Secor for transferring Kafka knowledge to S3 hosted on EC2. This was inflicting points after we tried to include new knowledge matters. My workforce decided that Secor fell quick by way of industry-standard recognition, ongoing growth, and help for rising knowledge codecs. I assumed accountability for this Secor setup, and I spearheaded its migration to Bedrock (an internally-developed Kubernetes framework). This transition simplified the addition of recent matters, harnessed the advantages of Kubernetes, and resulted in price financial savings by useful resource optimization.
To additional improve our knowledge infrastructure, we’re migrating from Secor to Kafka Join: a widely-adopted {industry} commonplace for streaming programs. Kafka Join gives out-of-the-box help for state administration, fault tolerance, and scaling. In contrast to Secor, Kafka Join operates as an abstraction and leverages connectors (executable JARs) throughout the Kafka-Join ecosystem. This strategic shift is transferring us in the direction of real-time streaming capabilities. The work aligns with our dedication to {industry} requirements, ongoing growth, and flexibility to evolving knowledge codecs.
My time with Slack’s Knowledge Infrastructure workforce has been a transformative journey. I’ve delved into mentorship, partaking tasks, and a tradition of steady studying. Slack’s dedication to mobility has enriched my progress by offering me the chance to work with all of the groups beneath knowledge and cross-organizational groups like Cloud and Observability. Once I returned from maternity depart, the workforce supported me by offering context about ongoing tasks and onboarding me again into the move. This expertise showcased Slack’s compassion by creating an surroundings the place skilled excellence is nurtured alongside empathy for private milestones.
Hello, I’m Nathalie Kaligirwa, a Senior Software program Engineer on the Metrics Platform & Governance workforce. My workforce builds scalable and standardized instruments to boost the information expertise. I joined in November 2021.
The previous two years have been eventful. I had a wholesome child, took six months of maternity depart, and labored on advanced tasks with a number of stakeholders and distributed groups. I aimed to steadiness private progress with work commitments by aligning duties with my power stage and busy schedule.
Metrics standardization is a significant ongoing initiative at Slack, and it encompasses tasks like Merlin, a framework for abstracting metrics and large tables creation. Throughout parenthood, I centered on particular elements inside this initiative that might be manageable throughout a time of elevated private calls for. This allowed me to contribute to the bigger imaginative and prescient of the challenge whereas balancing new guardian duties. For instance, I up to date a service to dynamically add new metrics created with Merlin and eradicated the necessity for handbook migration. I then switched gears to work on enhancing the search expertise. This set the groundwork for a brand new metadata service that gives lineage throughout a number of knowledge instruments and scalable search.
Whether or not I used to be consolidating metrics, studying to vary diapers, integrating knowledge providers, placing collectively a nursery, engaged on metadata providers, or navigating a compromised sleep schedule, I grew professionally and personally. Wanting again, I’m happy with my work and grateful for my workforce’s help.
As Slack’s Knowledge Engineering management, we’re dedicated to mentorship and offering alternatives to study and develop. The workforce can be dedicated to mobility — we encourage inside transfers for skilled progress and diversification of abilities. Slack’s steady studying philosophy not solely enriches particular person careers but additionally creates a dynamic and collaborative office.
We mannequin Slack’s worth of compassion in our help for workforce members taking maternity depart. The workforce is organized to make reintegration after a hiatus a seamless expertise. We genuinely care not nearly our teammate’s skilled efficiency but additionally their private milestones.
As leaders at Slack, we actively nurture an surroundings the place ladies in knowledge engineering thrive and attain their full potential. We do that by:
- Visibility: We amplify ladies’s voices by encouraging them to tackle management roles and communicate at conferences. We have fun their achievements and showcase them as position fashions by participation in {industry} occasions.
- Mentorship: We provide each formal and casual mentorship applications the place we join skilled ladies leaders with mentees for steerage and help. This fosters confidence, management abilities and a robust community of friends.
- Advocacy: We actively problem biases and advocate for honest practices in all points of our work surroundings.
- Empowerment: We help versatile work preparations, range and inclusion initiatives, and entry to coaching sources. We create a welcoming and inclusive area for all.
Embracing range: our dedication to inclusion
Our private experiences at Slack reveal the affect of those inclusion efforts. That is the Slack we envisioned: a spot the place concepts are heard, contributions are valued, and particular person journeys are fueled by a collaborative and supportive tradition. Our success is obvious within the rising variety of ladies selecting to affix our groups and within the legacy of compassion and inclusion we’re proud to domesticate.
Fascinated by becoming a member of our Knowledge Engineering workforce? Apply now
