Skip to content
goutham.dechineni

Freelance project

Heart Risk (retina + clinical)

Problem
Retinal blood vessels change with cardiovascular disease, and an eye photo is cheap to take, but a photo alone says little without the patient's clinical picture.
Approach
Two models scored separately and fused: a retinal model built on U-Net vessel features plus ImageNet EfficientNetB3 features, and a clinical model on patient records, combined 0.4 / 0.6 and explained by Gemini.
Result
A working pipeline from an eye photo and a short clinical form to one fused risk level with a plain-language explanation, built to show the approach rather than for clinical use.
My role
Freelance AI engineer (2024–2025). Built the system end to end for a client: data pipeline, models, training and app.
Stack
  • Python
  • TensorFlow / Keras
  • U-Net
  • EfficientNetB3
  • OpenCV
  • XGBoost
  • scikit-learn
  • Gemini
  • Streamlit
Links

01

Architecture

  1. Fundus image: a colour photo of the back of the eye, resized to a fixed input size.
  2. U-Net segments the blood vessels. The input is the green channel (where vessels have the most contrast), a CLAHE-enhanced copy of it, and a circular mask of the retina. It was trained on the public DRIVE retinal vessel dataset.
  3. Vessel features are measured from the segmentation mask: vessel density overall and per quadrant, vessel area and count, mean and maximum width, and tortuosity (how winding the vessels are).
  4. EfficientNetB3 with ImageNet weights supplies a vector of deep features per image. It is not fine-tuned on retinas, and the small projection layer on top was not trained either, so these are generic image features, not retina-specific ones.
  5. Retinal XGBoost combines the vessel features and the deep features into a retinal risk score. It was trained on images from a public diabetic-retinopathy set, with the retinopathy grade used as a stand-in for "at risk"; no dataset with retina photos and heart outcomes was available.
  6. Clinical form: age, sex, conditions (diabetes, high blood pressure, anaemia, smoking) and lab values such as ejection fraction, serum creatinine and serum sodium.
  7. Clinical XGBoost was trained on a small public heart-failure dataset of patient records, with death during follow-up as the target.
  8. Weighted fusion: the final score is 0.4 × retinal risk + 0.6 × clinical risk. The clinical side gets more weight: the retinal model learns from a stand-in label, so its score is the less direct evidence of heart risk.
  9. Gemini receives the scores and the risk factors that stood out (for example high vessel tortuosity or low ejection fraction) and writes an explanation and recommended next steps in plain language.
  10. Streamlit app: the report shows the overall risk level, the retinal and clinical scores, and the explanation.

02

Key decisions

  • Two separate models and a late fusion, instead of one joint model

    No available dataset had retina photos and clinical records for the same patients, so a joint model could not be trained. Two models, each trained on its own public data, made the system possible. The cost is that the 0.4 / 0.6 weights are set by hand from the two models' scores, not learned, and the two risk scores are not calibrated against each other.

  • Explicit vessel measurements next to deep features

    Density, width and tortuosity are features a clinician can recognise, and Gemini can name them in the explanation, which a vector of deep features does not allow. The cost is a second model to maintain, and the U-Net learned from a small labelled vessel dataset.

  • ImageNet features without fine-tuning

    With a small image set and a stand-in label, fine-tuning a large CNN would very likely overfit, so EfficientNetB3 is used as a fixed feature extractor. The cost is generic rather than retina-specific features, and the deep features the retinal model relies on most have no meaning that can be read off.

03

Results

  • Built the full pipeline end to end: vessel segmentation, vessel measurements, retinal and clinical risk models, weighted fusion and a Gemini-written explanation in a Streamlit app.
  • Each report names the risk factors that stood out, such as high vessel tortuosity or low ejection fraction, next to the fused risk level.
  • Trained only on small public datasets, one of them with a stand-in label, so no performance figures are claimed; it demonstrates the approach, not clinical fitness.

04

What I'd do next

  • Drop the follow-up-time column from the clinical model. It was the strongest feature, but it is only known after the outcome, so the clinical score is optimistic.
  • Train the projection layer on top of EfficientNetB3, or use a retina-specific pretrained encoder, instead of untrained layers over ImageNet features.
  • Learn the fusion weights and calibrate both risk scores on data that has retina photos and clinical records for the same patients.
  • Tune the clinical model's decision threshold so fewer high-risk patients are missed, since a missed patient costs more than a false alarm.