# Rethinking Bioinformatics > Solving the Challenges of Traditional Pipelines - URL: https://lokahq.github.io/tech-blog/rethinking-bioinformatics/ - Type: Blog article - Authors: Sara Oquendo (Bioinformatics Intern) - Published: 2026-01-05 - Updated: 2026-09-30 - Reading time: 8 min - Tags: Bioinformatics, Cloud Computing, Data Engineering, Scientific Workflows, Connectedlab - Originally published on Medium: https://medium.com/loka-engineering/rethinking-bioinformatics-fced54626712 --- **Written by Sara Oquendo, Bioinformatic intern** Bioinformatics workflows face a significant challenge: data is growing faster than our ability to process it with precision and traceability. This issue is especially problematic in today’s omics-driven research, where laboratories routinely handle terabytes of sequencing data and perform multi-step analyses involving alignment, variant calling, annotation, statistical modeling, and visualization. At the same time, computational processes are connected with digital laboratory systems such as LIMS, IoT-enabled sensors, and electronic lab notebooks, forming complex data ecosystems. As a result, both the scaling and heterogeneity of data have increased, making reproducibility and auditability harder to ensure and seamlessly integrate with cloud or hybrid infrastructures. Within this context, a common bottleneck in many labs occurs when scaling “traditional” or ad-hoc bioinformatics pipelines: long shell, Python, or R scripts that evolved over time, passed between teams and adapted to specific instrumentation or one time projects. For example, a sequencing facility increases its workload from a few dozen to several hundred genomes per month, often relying on manual file transfers, untracked software updates and computing systems configured in different ways. These factors collectively introduce delays, inconsistencies, and errors.Taken together, these issues only become more visible as data grows, making clear some central limitations of traditional pipelines: limited flexibility, poor reusability, constrained scalability, and insufficient traceability (Via Scientific, n.d). At the same time newer challenges are beginning to determine how modern bioinformatics pipelines operate. As labs scale up and migrate analyses onto the cloud and hybrid platforms, issues arise with standardizing workflows and alternating massive datasets between these systems. These processes become even more complex in regulated settings like healthcare and biotechnology, where nearly 40% of organizations now require end-to-end encryption and audit trails before trusting a cloud pipeline (MarketGrowthReports, n.d.). At the same time, the need for accurate data has never been higher, as recent reports indicate that approximately one-third of published results contain errors attributable to faulty inputs and data processing (Omni-Inc., n.d.). Beyond infrastructure challenges, the scientific work itself is becoming harder to manage. More and more labs are working with machine learning models and combining different omics analyses, but of course working with these tools together is tough since different data formats and systems don’t match with each other; plus many AI models still operate like black boxes which raises concerns among clinical teams and regulators. Labs also face everyday issues when their instruments, software, data platforms, and overall work are stored in different places, which necessitates manual file handling, formatting and double-checking. Taken together, these technical, organizational and regulatory challenges have sparked a clear shift towards more flexible and connected solutions like modular architecture and integrated lab systems — frameworks that are essential for making pipelines more flexible, reliable, scalable, reproducible, and traceable (Omni-Inc., n.d.; MarketGrowthReports, n.d.). ## The Challenge: Building a Modern Bioinformatics Pipeline When designing a scalable, adaptable, and reproducible bioinformatics pipeline that stays transparent from data generation to final analysis, the main question becomes less about the tools themselves and more about the architecture. The following example demonstrates how modular architectures and connected-lab principles can collectively solve practical challenges in omics research. ![End-to-End Example Cloud Architecture for Triggering Omics Pipeline Execution Using AWS Services.](https://lokahq.github.io/tech-blog/blog/rethinking-bioinformatics/1-r7-dEzdD0MIzRi1FaBZ9eQ.webp) **Figure 1. End-to-End Example Cloud Architecture for Triggering Omics Pipeline Execution Using AWS Services. **This diagram shows the end-to-end AWS cloud architecture for a modular bioinformatics workflow. (1) Public genomic data from repositories like UniProt and NCBI is (2) ingested into an Amazon S3 landing bucket. (3) Amazon EventBridge detects S3 uploads and triggers (4) AWS Step Functions, the central orchestrator. Step Functions coordinates the Nextflow workflow execution, supporting two parallel options: (5) AWS HealthOmics, a managed omics-optimized environment, or (6) AWS Batch, a flexible compute service. Both pull containerized components from Amazon ECR. (7) Results are returned to S3 for downstream analysis, generating insights and visualizations. In this example, I explore two cloud-native approaches for running bioinformatics pipelines on AWS, AWS HealthOmics and AWS Batch, using a shared workflow. While both approaches produce the same outputs, their implementations, advantages, and trade-offs differ depending on the use case. The pipeline follows the typical structure of many omics workflows: raw data ingestion, quality control, alignment, variant calling, and reporting. Instead of packaging these stages into a single monolithic script, each step was designed as an independent, reusable module. All components run inside containers and are orchestrated with Nextflow, a workflow manager that supports cloud execution, scalable parallelization, and reproducibility (Di Tommaso et al., 2017; Nextflow, n.d.). ### From a Script Pile to a Modular System Traditional pipelines are often inflexible, one massive script that processes data from start to finish. They can be quick to implement but difficult to maintain or scale. Modular design changes that logic by breaking the process into small, containerized steps with clear inputs and outputs. Each part becomes replaceable, testable, and reusable (Leipzig, 2017). Tools like Docker or Singularity handle the environment. Workflow engines like Nextflow or Snakemake manage the flow. And cloud services like AWS handle the scale. Suddenly we can now mix, match, rerun, or extend parts of the pipeline without rewriting the whole thing (Nextflow, n.d.). ### Why Modularity Works (Especially in Bioinformatics) Bioinformatics evolves rapidly; new tools, formats and practices appear regularly. A modular design easily adjusts to these changes because: - **It adapts:** When testing a new variant caller or aligner, swapping the module rather than the entire pipeline ensures that the updates are localized and non-disruptive. - **It scales:** Modular pipelines parallelize naturally. While one sample is aligning, another is already trimming. AWS Batch or HealthOmics can achieve faster runs and smarter compute use (Amazon Web Services, n.d.). - **It reuses:** Common steps like QC, alignment, and trimming can easily be reused across datasets. As an example, the [nf-core](https://nf-co.re/) community is built on this idea (nf-core community, n.d.). - **It tracks everything:** Each module runs in a container, with logging, versioning, and defined outputs. That makes results traceable, audit-friendly, and reproducible Musts in regulated or collaborative settings (Wilkinson, 2016). This setup also simplified the implementation. The pipeline itself remained unchanged between AWS Batch and HealthOmics. Only the execution layer differed, which makes it easier to choose the execution model that best fits the specific use case. ### Common Tools Supporting Modularity Several workflow managers are designed for modular pipelines, each offering various advantages and disadvantages: - **Nextflow:** One of the most widely adopted tools in bioinformatics supports containers, cloud, and parallelism. In addition, it has a strong web-based community and cloud-native support. - **Snakemake:** Python-based, ideal for academic settings. Allows modular rule reuse, integrates well with local and cloud clusters. - **Galaxy:** User- friendly visual interface, popular in industry and core labs. Modular, though less flexible for custom logic. - **Toil/Cromwell:** Built for large-scale pipelines using WDL or CWL. Focused on scale and reproducibility. Choosing the right tool depends on the context, team expertise, infrastructure, and how complex or reusable the workflow needs to be. However, what they all share is the ability to turn analysis pipelines into structured, maintainable systems. Usually, as data volume increases alongside its project scaling, using one of these tools becomes more of a necessity instead of a convenience (Leipzig, 2017). ### When Not to Modularize Modularizing every single workflow isn’t always the best approach. For small, quick analyses or experiments that change every day, setting up containers and workflow configs can be an unnecessary effort. In those cases, simple scripting might be the more practical choice. But when scaling, reusing or sharing a workflow across a team, modular design becomes a long-term choice by reducing re-work, improving clarity, and letting pipelines evolve without breaking everything apart. ## Connected Labs: Where the Lab Meets the Cloud Another defining characteristic of the project is its integration to actual laboratory workflows. While the data used came from open biological repositories, the architecture was designed to mirror a lab-to-cloud ecosystem, an environment where the instruments that generate raw data, systems that manage metadata, and compute pipelines that analyze results are all interconnected. In this setup, after a sample is processed at the bench, its digital counterpart automatically moves through ingestion, quality checks, and analysis, with every step logged and versioned for full traceability from start to finish. ### What Is a Connected Lab? A connected lab is a digitally integrated environment where physical lab instruments, data management systems, and analytical tools operate as one coordinated system. Technologies like cloud computing, Internet of Things (IoT) enabled devices, artificial intelligence (AI) and LIMS/ELN platforms monitor and control experiments centrally, with data automatically collected and accessible in real time (LabHorizons, 2024). ### Three Ways Connected Labs Improve Scientific Workflows Disconnected systems create bottlenecks. For example, technicians manually transferring files from sequencers to storage, researchers retyping sample metadata into analysis tools, or delays caused by waiting for someone to trigger the next step in the workflowcan all slow down progress, increase the risk of error, waste time and money, and make it harder to keep data organized and reproducible. Connected labs are designed to address these gaps through tighter integration, automation, and traceability (LabHorizons, 2024): 1. In the case of automation and workflow triggers, routine steps that once required human involvement like launching a pipeline or uploading results can now be fully automated. In the example, uploading new sequencing data to an S3 landing bucket automatically triggered the analysis workflow via AWS EventBridge. The connection between data entry and pipeline launch was real-time and hands-free, simulating how lab instruments can trigger downstream analysis, reporting, or alerts without technician intervention. 2\. Traceability in connected labs is a foundational layer. Each stage of the workflow, from sample acquisition to final output, is tied to structured metadata that can include sample identifiers, instrument logs, reagent batch numbers, and protocol versions (LabHorizons, 2024). Also, containers were versioned in Amazon ECR, while execution metadata such as runtime configuration and performance logs were captured via Amazon CloudWatch. These records copy the kind of digital audit trail that tools like LIMS or ELNs provide in physical lab environments, ensuring that: - Each protocol is properly versioned. - Data inputs and outputs are fully documented. - A complete, verifiable audit trail is maintained. This tracking is essential for simplifying troubleshooting, and maintaining reproducibility and data integrity, especially in research that demands validation, peer review, or regulatory oversight. 3\. When it comes to compliance, most laboratory conditions are controlled by strict standards. Whether in clinical research, pharmaceutical development, or regulatory certification, there’s a constant need to show that processes are applied consistently and results are trustworthy. Connected systems ease this pressure by embedding compliance directly into day-to-day operations. They enable: - Automated logging of actions, versions, and runtime details - Role-based access control and secure data retention - Enforced use of validated, standardized workflows This approach to compliance was reflected in the example through the use of AWS HealthOmics, a managed execution environment that provides regulatory alignment out of the box. Combined with CloudWatch monitoring, the setup ensured that every run was reliable, traceable, and free from manual follow-up (Amazon Web Services, n.d.). ### Managing Data Scale, Complexity, and Compliance in Modern Laboratories Modern laboratories are encountering challenges that make digital connectivity not just beneficial, but indispensable. As research outputs grow, producing massive volumes of omics data while simultaneously relying on increasingly complex instrumentation, labs must ensure accuracy, traceability, and compliance at every step. Connected-lab systems make this possible by integrating data streams, instruments, and computational workflows so that information flows seamlessly and can be processed in real time. This level of integration enables faster, better-informed decisions; reduces bottlenecks; and strengthens reliability. Automated monitoring can flag issues, such as equipment malfunctions or potential sample contamination, before they escalate into costly setbacks. As highlighted by _LabManager_, organizations that invest in connected and automated laboratory environments are better positioned to improve workflow efficiency, maintain reproducibility, and control operational costs. Over time, these benefits help reduce the impact of short-term transformation challenges in today’s competitive research environment. ## References Amazon Web Services. (n.d.). _AWS HealthOmics._ Retrieved December 4, 2025, from Di Tommaso, P., Chatzou, M., Floden, E. _et al._ Nextflow enables reproducible computational workflows. _Nat Biotechnol_ 35, 316–319 (2017). Jeremy Leipzig, A review of bioinformatic pipeline frameworks, _Briefings in Bioinformatics_, Volume 18, Issue 3, May 2017, Pages 530–536, LabHorizons. (2024, May). _What is a connected lab and how is it helpful?_ LabHorizons. Retrieved December 4, 2025, from LabManager. (n.d.). _Unlocking efficiency through lab transformation._ LabManager. Retrieved December 4, 2025, from MarketGrowthReports. (n.d.). _Bioinformatics Cloud Platform Market Report._ MarketGrowthReports. Retrieved December 4, 2025, from Nextflow. (n.d.). _Nextflow documentation._ Retrieved December 4, 2025, from [https://www.nextflow.io/](https://www.nextflow.io/?utm_source=chatgpt.com) nf-core community. (n.d.). _nf-core._ Retrieved December 4, 2025, from Omni-Inc. (n.d.). _Garbage in, garbage out — dealing with data errors in bioinformatics._ Omni-Inc. Retrieved December 4, 2025, from Via Scientific. (n.d.). _How modular pipelines are changing bioinformatics._ Via Scientific. Retrieved December 4, 2025, from [viascientific.com](http://viascientific.com/) Wilkinson, M., Dumontier, M., Aalbersberg, I. _et al._ (2016). The FAIR Guiding Principles for scientific data management and stewardship. _Sci Data_ 3, 160018 (2016).