The world of deep studying is quickly evolving, with pretrained fashions turning into more and more essential for a variety of duties. Keras, recognized for its user-friendly API and concentrate on accessibility, has been on the forefront of this motion with specialised libraries like KerasNLP for text-based fashions and KerasCV for pc imaginative and prescient fashions.
Nonetheless, as fashions more and more blur the traces between modalities – consider highly effective chat LLMs with picture inputs or imaginative and prescient duties leveraging textual content encoders – sustaining these separate domains is much less sensible. The division between NLP and CV can hinder the event and deployment of really multimodal fashions, resulting in redundant efforts and a fragmented person expertise.
To handle this, we’re excited to announce a serious evolution within the Keras ecosystem: KerasHub, a unified, complete library for pretrained fashions, streamlining entry to each cutting-edge NLP and CV architectures. KerasHub is a central repository the place you’ll be able to seamlessly discover and make the most of state-of-the-art fashions like BERT for textual content evaluation alongside EfficientNet for picture classification, all inside a constant and acquainted Keras framework.
A unified developer expertise
This unification not solely simplifies mannequin discovery and utilization but in addition fosters a extra cohesive ecosystem. With KerasHub, you’ll be able to leverage superior options like easy mannequin publishing and sharing, LoRA fine-tuning for resource-efficient adaptation, quantization for optimized efficiency, and strong multi-host coaching for tackling large-scale datasets, all relevant throughout various modalities. This marks a major step in the direction of democratizing entry to highly effective AI instruments and accelerating the event of progressive multimodal functions.
First steps with KerasHub
Let’s get began by putting in KerasHub in your system. From there, you’ll be able to discover the intensive assortment of readily available models and completely different implementations of well-liked architectures. You may then be prepared to simply load and incorporate these pre-trained fashions into your individual tasks and fine-tune them for optimum efficiency in accordance with your particular necessities.
Putting in KerasHub
To put in the newest KerasHub launch with Keras 3, merely run:
$ pip set up --upgrade keras-hub
Now you can begin exploring the accessible fashions. The usual surroundings setup to start out working with Keras 3 does not change in any respect to start out utilizing KerasHub:
import os
# Outline the Keras 3 backend you need to use - "jax", "tensorflow" or "torch"
os.environ["KERAS_BACKEND"] = "jax"
# Import Keras 3 and KerasHub modules
import keras
import keras_hub
Utilizing pc imaginative and prescient and pure language fashions with KerasHub
Now you might be prepared to start out with KerasHub to entry and use the fashions accessible at Keras 3 ecosystem. Some examples beneath:
Gemma
Gemma is a group of cutting-edge, but accessible, open fashions developed by Google. Leveraging the identical analysis and know-how behind the Gemini fashions, Gemma’s base fashions excel at varied textual content era duties. These embrace answering questions, summarizing info, and interesting in logical reasoning. Moreover, they are often personalized to handle particular wants.
On this instance you utilize Keras and KerasHub to load and begin producing contents utilizing Gemma2 2B parameters. For extra particulars about Gemma variants, check out the Gemma model card at Kaggle.
# Load Gemma 2 2B preset from Kaggle fashions
gemma_lm = keras_hub.fashions.GemmaCausalLM.from_preset("gemma_2b_en")
# Begin producing contents with Gemma 2 2B
gemma_lm.generate("Keras is a", max_length=32)
PaliGemma
PaliGemma is a compact, open mannequin that understands each photographs and textual content. Drawing inspiration from PaLI-3 and constructed on open-source elements just like the SigLIP vision model and the Gemma language model, PaliGemma can present detailed and insightful solutions to questions on photographs. This permits for a deeper understanding of visible content material, enabling capabilities similar to producing captions for photographs and brief movies, figuring out objects, and even studying textual content inside photographs.
import os
# Outline the Keras 3 backend you need to use - "jax", "tensorflow" or "torch"
os.environ["KERAS_BACKEND"] = "jax"
# Import Keras 3 and KerasHub modules
import keras
import keras_hub
from keras.utils import get_file, load_img, img_to_array
# Import PaliGemma 3B fantastic tuned with 224x224 photographs
pali_gemma_lm = keras_hub.fashions.PaliGemmaCausalLM.from_preset(
"pali_gemma_3b_mix_224"
)
# Obtain a check picture and put together it for utilization with KerasHub
url = 'https://storage.googleapis.com/keras-cv/fashions/paligemma/cow_beach_1.png'
img_path = get_file(origin=url)
img = img_to_array(load_img(image_path))
# Create the immediate with the query in regards to the picture
immediate = 'reply the place is the cow standing?'
# Generate the contents with PaliGemma
output = pali_gemma_lm.generate(
inputs={
"photographs": img,
"prompts": immediate,
}
)
For extra particulars in regards to the accessible pre-trained fashions on Keras 3, take a look at the list of models in Keras on Kaggle.
Stability.ai Secure Diffusion 3
You have got the pc imaginative and prescient fashions accessible for utilization too. For instance you should use stability.ai Stable Diffusion 3 with KerasHub:
from PIL import Picture
from keras.utils import array_to_img
from keras_hub.fashions import StableDiffusion3TextToImage
text_to_image = StableDiffusion3TextToImage.from_preset(
"stable_diffusion_3_medium",
top=1024,
width=1024,
dtype="float16",
)
# Generate photographs with SD3
picture = text_to_image.generate(
"{photograph} of an astronaut using a horse, detailed, 8k",
)
# Show the generated picture
img = array_to_img(picture)
img
For extra particulars in regards to the accessible pre-trained pc imaginative and prescient fashions on Keras 3, test the list of models in Keras.
What modifications for KerasNLP builders?
The transition from KerasNLP to KerasHub is an easy course of. It solely requires updating the import statements from keras_nlp to keras_hub.
Instance: Beforehand in the event you have been importing keras_nlp to make use of a BERT mannequin like beneath
import keras_nlp
# Load a BERT mannequin
classifier = keras_nlp.fashions.BertClassifier.from_preset(
"bert_base_en_uncased",
num_classes=2,
)
Modify the import, and you might be able to go along with KerasHub:
import keras_hub
# Load a BERT mannequin
classifier = keras_hub.fashions.BertClassifier.from_preset(
"bert_base_en_uncased",
num_classes=2,
)
What modifications for KerasCV builders?
In case you are a present KerasCV person, updating to KerasHub provides you these advantages:
- Simplified Mannequin Loading: KerasHub affords a constant API for loading fashions, which may simplify your code in the event you’re working with each KerasCV and KerasNLP.
- Framework Flexibility: In the event you’re considering exploring completely different frameworks like JAX or PyTorch, KerasHub makes it simpler to make use of KerasCV and KerasNLP fashions with them.
- Centralized Repository: Discovering and accessing fashions is less complicated with KerasHub’s unified mannequin repository and is the place new architectures can be added sooner or later.
How one can adapt my code to KerasHub?
Fashions
KerasCV fashions are at present being ported to KerasHub. Whereas most are already accessible, a number of are nonetheless a piece in progress. Please notice that the Centerpillar mannequin is not going to be ported. It is best to be capable of use any imaginative and prescient mannequin in KerasHub with:
import keras_hub
# Load a mannequin utilizing preset
Mannequin = keras_hub.fashions.<model_name>.from_preset('preset_name`)
# or load a customized mannequin by specifying the spine and preprocessor
Mannequin = keras_hub.fashions.<model_name>(spine=spine, preprocessor=preprocessor)
KerasHub introduces thrilling new options for KerasCV builders, providing higher flexibility and expanded capabilities. It contains:
In-built preprocessing
Every mannequin is accompanied by a bespoke preprocessor that addresses routine duties together with resizing, rescaling, and extra, streamlining your workflow.
Previous to this, the enter preprocessing was carried out manually previous to offering the inputs to the mannequin.
# Preprocess inputs for instance
def preprocess_inputs(picture, label):
# Resize rescale or do extra preprocessing on inputs
return preprocessed_inputs
spine = keras_cv.fashions.ResNet50V2Backbone.from_preset(
"resnet50_v2_imagenet",
)
mannequin = keras_cv.fashions.ImageClassifier(
spine=spine,
num_classes=4,
)
output = mannequin(preprocessed_input)
At the moment, the duty fashions’ preprocessing is built-in inside the established presets. The inputs endure preprocessing, the place pattern photographs endure resizing and rescaling inside the preprocessor. The preprocessor is an intrinsic part of the duty mannequin. However, one has the choice to make the most of a customized preprocessor.
classifier = keras_hub.fashions.ImageClassifier.from_preset('resnet_18_imagenet')
classifier.predict(inputs)
Loss features
Much like augmentation layers, loss features beforehand in KerasCV are actually accessible in Keras by way of keras.losses.<loss_function>. For instance, in case you are at present utilizing FocalLoss function:
import keras
import keras_cv
keras_cv.losses.FocalLoss(
alpha=0.25, gamma=2, from_logits=False, label_smoothing=0, **kwargs
)
You simply want to regulate your loss perform definition code to make use of keras.losses as a substitute of keras_cv.losses:
import keras
keras.losses.FocalLoss(
alpha=0.25, gamma=2, from_logits=False, label_smoothing=0, **kwargs
)
Get began with KerasHub
Dive into the world of KerasHub at this time:
Be a part of the Keras neighborhood and unlock the ability of unified, accessible, and environment friendly deep studying fashions. The way forward for AI is multimodal, and KerasHub is your gateway to it!
