DZone Spotlight

When considering whether to use classes in Python code, it's important to weigh the benefits and contexts where they are most appropriate. Classes are a key feature of object-oriented programming (OOP) and can provide clear advantages in terms of organization, reusability, and encapsulation. However, not every problem requires OOP. When to Use Classes Encapsulation and Abstraction Encapsulation. Classes allow you to bundle data (attributes) and methods (functions) that operate on the data into a single unit. This helps in keeping related data and behaviors together.Abstraction. Classes can hide the complex implementation details and expose only the necessary parts through public methods. Reusability Reusable components. Classes can be represented multiple times, allowing you to create reusable components that can be used in different parts of your application. Inheritance and Polymorphism Inheritance. Classes allow you to create new classes based on existing ones, reusing the code and reducing redundancy.Polymorphism. Classes enable you to define methods in such a way that they can be used interchangeably, improving flexibility and integration. Stateful Function Using Classes Using a class is a more explicit way to manage state. Use classes when the stateful function requires multiple methods. Python class Counter def__init__(self): self.count =0 def increment(self): self.count =1 return self.count # Create a counter instance counter1 = Counter() print(counter1.increment()) #Output:1 print(counter1.increment()) #Output:2 print(counter1.increment()) #Output:3 # Create another counter instance counter2 = Counter() print(counter2.increment()) #Output:1 print(counter2.increment()) #Output:2 When Not to Use Classes Simple Scripts and Small Programs For small scripts and simple programs, using functions without classes can keep the code straightforward and easy to understand. Stateless Functions If your functions do not need to maintain any state and only perform operations based on input arguments, using plain functions is often more appropriate. Best Practices for Using Classes Naming Conventions Use CamelCase for class names.Use descriptive names that convey the purpose of the class. Single Responsibility Principle Ensure each class has a single responsibility or purpose. This makes them easier to understand and maintain. Best Practices for Writing Functions Function Length and Complexity The functions should be small, with the focus on a single item.Aim for functions to be no longer than 50 lines; if they are longer, consider refactoring. Pure Functions Write pure functions where possible. Its output is determined only by its input and has no side effects. It always produces the same output for the same input. It has several benefits, including easier testing and debugging, as well as better code maintainability and reusability. They also facilitate parallel and concurrent programming since they don't rely on the shared state or mutable data. Python def add(a,b): """Add two numbers.""" return a + b #Usage result = add(4,2) print(result) # Output:6 Impure Functions It relies on external state and produces side effects, which makes its behavior unpredictable. Python total = 0 def add_to_total(value): """Add a value to the global total""" global total total += value #Usage: add_to_total(4) print(total) #Output: 4 add_to_total(2) print(total) #Output: 6 Documentation Use docstrings to document the purpose and behavior of functionsInclude type hints to specify the expected input and output types Static Methods It is used when you want a method to be associated with a class rather than an instance of the class. Below are a few scenarios. Grouping utility functions. It is a group of related functions that logically belong to a class but don't depend on instance attributes or methods.Code organization. It helps in organizing the code by grouping related functions with in a class, even if they don't operate on instance-level data.Encapsulation. It clearly communicates that they are related to the class, not to any specific instance, which can enhance code readability. DateUtils.py Python from datetime import datetime class Dateutils: @staticmethod def is_valid_date(date_str, format="%Y-%m-%d"): """ checks if a string is a valid date in the specified format. Args: date_str(Str): the date strig to validate. format(str, optional): the date format to use (default "%Y-%m-%d"). Returns: bool: true if the date is valid, false otherwise. """ try: datetime.strptime(date_str, format) return True except ValueError: return False @staticmethod def get_days_difference(date1_str, date2_str, format ="%Y-%m-%d"): """ calculates number of days between two dates Args: date1_str(Str): the first date string. date2_str(Str): the second date string. format(str, optional): the date format to use (default "%Y-%m-%d"). Returns: int: the number of days between two dates. """ date1 = datetime.strptime(date1_str, format) date2 = datetime.strptime(date2_str, format) return abs(date2 - date1).days) # abs() is used for non negative difference #usage valid_date = DateUtils.is_valid_date("2025-03-07") print(f"Is'2025-03-07' is a valid date? {valid_date}") #output:True days_diff = DateUtils.get_days_difference("2025-02-01","2025-03-01") print(f"Days between '2025-02-26' and '2025-03-01': {days_diff}" Logger It's a built-in logging module that is a flexible and powerful system for managing application logs. It is essential for diagnosing issues and monitoring application behavior. The RotatingFileHandler is a powerful tool for managing log files. It allows you to create log files that automatically rotate when they reach a certain size. log_base_config.py Python import logging from logging.handlers import RotatingFileHandler def setup_logging(): #create logger logger = logging.getLogger() logger.setLevel(logging.INFO) #create a rotating file handler rotating_handler = RotatingFileHandler('example.log', maxBytes=1024, backupcount =30) rotating_handler.setLevel(logging.INFO) #create a formatter and add it to the handler formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') rotating_handler.setFormatter(formatter) #add rotating file handler to the logger logger.addHandler(rotating_handler) logger_impl.py Python import logging from log_base_config import setup_loggimg setup_logging() logger = logging.getLogger("logger_example") def call_logger(numerator:int,denominator:int): logger.info("call_logger(): start") #log some messages for i in range(32): logger.info(f'This is a message {i}') try: result = numerator / denominator logger.info("Division successful: %d / %d = %f", numerator, denominator, result) return result except ZeroDivisionError: logger.error("Failed to perform division due to a zero division error.") return None logger.info("call_logger(): end") if __name__ == "__main__": call_logger(12,0) Do Not Consume Exception It is catching an exception without taking appropriate action such as logging the error or raising the exception again. This can make debugging difficult because the program fails silently. Instead you should handle exception in a way that provides useful information and allows the application to either recover or fail gracefully. Here's how you can properly handle exceptions without consuming them: Ensure the exception details are logged.Provide user-friendly feedback or take corrective actions.If appropriate, raise the exception again after logging it. Below is an example showing how to properly handle and log a ZeroDivisionError without consuming the exception. Python import logging from log_base_config import setup_loggimg setup_logging() logger = logging.getLogger("logger_example") def divide_numbers(numerator, denominator): try: result = numerator / denominator logger.info("Division successful: %d / %d = %f", numerator, denominator, result) return result except ZeroDivisionError as e: logger.error("Division by zero error: %d / %d", numerator, denominator) logger.exception("Exception details:") #raise the exception again after logging raise # test the function num = 10 denon = 0 try logger.info("Attempting to divide %d by %d", num, denom) result = divide_number(num, denom) except ZeroDivisionError: logger.error("Failed to perform division due to a zero division error.") Conclusion By following these techniques, one can ensure that exceptions are handled transparently, aiding in debugging and maintaining robust application behavior.

In this article, we’ll provide some background about Word DOCX forms and discuss how form values (i.e., form responses) can be programmatically extracted from completed DOCX forms in Java. We’ll discuss the specific way in which form responses are stored in the DOCX file structure and how that impacts a programmatic process aimed at retrieving them. Towards the end of this article, we’ll learn about a few different APIs we can use to build an expedited DOCX form-value retrieval workflow in Java. Background: DOCX Forms Much like PDF, the Word DOCX format provides the option to build custom form fields into a template form document. This option is available when developer mode is enabled in a Word DOCX document; administrators can implement a range of customizable fields, including rich and plain text boxes, checkboxes, dropdown menus, date selections, and more. Administrators can share DOCX forms with various recipients to collect information; when responses are submitted, a copy of the completed form can be saved, and its response contents can be extracted (either manually or via automated processes). How Forms Are Structured in DOCX File Format Like all Open Office XML file types, DOCX files are structured as a series of zipped (ZIP archived) XML files. This is an exceptionally developer-friendly setup (especially when compared with the pre-2007 binary Office file structure). The Open Office XML structure allows us to reach into Office files and follow a standardized XML path to extract just about any content we need independently from other document content. In any DOCX file archive, we can find all the Word-specific content we need in the word/folder within the document.xml file (this includes text, formatting, and embedded elements — including form fields). DOCX form fields are specifically represented as “content controls”, and these can be found in <w:sdt> (Structured Document Tag) elements in the word/document.xml file. Each <w:sdt> contains a nested <w:sdtPr> element defining the properties and settings of a given content control, a <w:docPartObj> defining the “type” or “gallery” of the content control, and a <w:docPartGallery w:val= “”/>, where w:val= is set to the specific type of content the content control in question expects (e.g., w:val=“Text” for a text field). The <w:sdtContent> element contains the actual content in any given control, i.e., the placeholder text or the user-entered data. Automating Form Content Retrieval in Java Given what we know about the DOCX file structure, getting DOCX form responses is relatively straightforward. Our programmatic process must: Unzip the DOCX fileNavigate to the word/ folderFind the document.xml fileSelect <w:sdtContent> The only thing standing in our way is the implementation of actual Java code to satisfy that process. While coding this workflow from scratch might make for a fun Java side-project, it’s a bit less practical under the time constraints of most production environments. Thankfully, we can lean on a few different API solutions to expedite our own implementation. Open-Source API for DOCX Form Content Retrieval As far as open-source options go, Apache POI remains the strongest recommendation (we’ve suggested this library a lot in past articles). With Apache POI, we can represent our DOCX content with the XWPFDocument class, retrieve <w:sdt> elements with the XWPFSDT class, check for different field types in the resulting list (e.g., text, checkbox, checkdown, etc.), and call various methods like isChecked() (for checkbox fields) and getText() (for text-based fields) to extract form submission values. Fully Realized Web API Solution If we’re looking for a quick, plug-and-play option with less documentation-reading involved (not to mention less memory consumption in our environment), we can use a free web API to get the job done instead. This option will abstract file processing away from our environment and return a simple, neatly structured response body containing an array of all the content control values present in a given DOCX form. Below, we’ll walk through each step required to call this API. We’ll start by installing the SDK. If we’re installing with Maven, we’ll add the following reference to the repository in pom.xml: XML <repositories> <repository> <id>jitpack.io</id> <url>https://jitpack.io</url> </repository> </repositories> Next, we’ll add the following reference to the dependency in pom.xml: XML <dependencies> <dependency> <groupId>com.github.Cloudmersive</groupId> <artifactId>Cloudmersive.APIClient.Java</artifactId> <version>v4.25</version> </dependency> </dependencies> If we’re installing with Gradle, we’ll add the following to our root build.gradle (at the end of repositories): Groovy allprojects { repositories { ... maven { url 'https://jitpack.io' } } } And we’ll then add the dependency in build.gradle: Groovy dependencies { implementation 'com.github.Cloudmersive:Cloudmersive.APIClient.Java:v4.25' } With installation out of the way, we’ll add the import classes at the top of our file: Java // Import classes: //import com.cloudmersive.client.invoker.ApiClient; //import com.cloudmersive.client.invoker.ApiException; //import com.cloudmersive.client.invoker.Configuration; //import com.cloudmersive.client.invoker.auth.*; //import com.cloudmersive.client.EditDocumentApi; Next, we’ll implement some code to handle configuration and authorization (we’ll need a free API key to authorize our requests): Java ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: Apikey ApiKeyAuth Apikey = (ApiKeyAuth) defaultClient.getAuthentication("Apikey"); Apikey.setApiKey("YOUR API KEY"); // Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null) //Apikey.setApiKeyPrefix("Token"); Finally, we’ll create an instance of the EditDocumentAPI, create a file input, and implement a try/catch block to make our API call: Java EditDocumentApi apiInstance = new EditDocumentApi(); File inputFile = new File("/path/to/inputfile"); // File | Input file to perform the operation on. try { GetDocxGetFormFieldsResponse result = apiInstance.editDocumentDocxGetFormFields(inputFile); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling EditDocumentApi#editDocumentDocxGetFormFields"); e.printStackTrace(); } We can expect our response to follow the below model: JSON { "Successful": true, "ContentControls": [ { "Value": "string" } ], "HandlebarFormFields": [ { "FormFieldTitle": "string" } ] } We’ll notice two separate arrays in the above model — one for “ContentControls” and one for “HandlebarFormFields”. We’re only interested in the “ContentControls” values in the context of this article, but it’s worth noting that the “HandlebarFormFields” array will return any form fields structured as {{enter_value} in our DOCX form. The double curly bracket structure is referred to as a handlebar form; this option is sometimes used for programmatic text-replacement workflows (e.g., when web form responses are passed to a DOCX file and content controls are unnecessary). After we return our content control value array, we can iterate values over any data type and dynamically store response data. Conclusion In this article, we discussed DOCX forms and how form values (responses) can be programmatically accessed from the DOCX file structure. We suggested open-source and non-open-source API solutions to simplify the implementation of form value retrieval in a Java environment.

By Brian O'Neill CORE

Amazon Nova is a new generation of state-of-the-art (SOTA) foundation models (FMs) that deliver frontier intelligence and industry-leading price performance, released on December 24. It is available exclusively on Amazon Bedrock. As this model is quite new, there are not a lot of examples of its use, especially in Java. As of this writing, the official AWS docs contain only Python code examples, and there are a couple. Today, we are going to build a Java application that initiates video generation using a text prompt, using Amazon Bedrock API and Amazon Nova Foundational model. The resulting video will be saved in AWS S3. Note: Please be aware that executing this application may cost you some money for running AWS Bedrock. Step 1: Generate AWS Keys and Enable the Foundational Model to Use If you don’t have an active AWS access key, do the following steps (copied and pasted from this SOF thread): Go to: http://aws.amazon.com/.Sign up and create a new account (they'll give you the option for a 1-year trial or similar).Go to your AWS account overview.The account menu is in the upper right (it has your name on it).sub-menu: Security Credentials. After you have your keys generated, you should choose and enable the foundational model in Bedrock. Go to Amazon Bedrock, and from the Model Access menu on the left, configure access to the Nova Reel model. Please be aware that, as of the writing of this article, Amazon Nova models are available in Amazon Bedrock only in the US East (N. Virginia) AWS Region. Step 2: Install and Configure AWS CLI AWS Java SDK uses the AWS CLI configuration to grab user credentials (the access key we configured in step 1). To avoid providing keys directly in code, it is a better (and safer) approach to configure AWS CLI first and then start writing code. If you don’t have AWS CLI configured, complete the following steps: Download and install the AWS CLI.Configure the CLI with AWS configure. During the configuration process, paste the key information you received during step one. As a default region, I suggest setting us-east-1, as the Amazon Nova Reel model is only available in this region. Step 3: Create an S3 Bucket Our video will not be created automatically; rather, the API call will just trigger a new execution. The result will be generated in a couple of minutes and stored in the S3 bucket. As part of the configuration, we will have to provide an S3 bucket where this video should be stored. When video generation is complete, the video is stored in the Amazon S3 bucket we specified in this step. Amazon Nova creates a folder for each invocation ID. This folder contains the manifest.json and output.mp4 files created by the video generation request. Creating an S3 bucket is quite a straightforward process, but if you need help, please find additional instructions here. Remember the bucket name you create, as we will need it in the next steps. Because Amazon Bedrock writes a file to an Amazon S3 bucket on your behalf, the AWS role that you use needs permissions configured to allow the appropriate Amazon Bedrock and Amazon S3 actions and the s3:PutObject action. The minimum action permissions required to generate a video are: Plain Text bedrock:InvokeModel s3:PutObject However, AWS recommends the following additional actions so you can track the status of video generation jobs: Plain Text bedrock:GetAsyncInvoke bedrock:ListAsyncInvokes Step 4. Setting Up Dependencies As Amazon Nova is a quite new foundational model and its introduction caused some extension of existing interfaces provided by AWS SDK, we may not be able to use old versions of Bedrock SDK. As of writing, the latest version of the AWS Bedrock runtime is 2.30.35 and this is the version I’m going to use. Make sure you are adding this dependency in your pom.xml if you are using Maven. For Gradle, the syntax will be different, but ensure you are using the same or a later version of the Bedrock dependency. XML <dependencies> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>bedrockruntime</artifactId> <version>2.30.35</version> </dependency> </dependencies> Step 5: Writing Code Below, you can find the application code. As the main purpose of this article is to explore the way to integrate Java with the AWS Bedrock/Nova Reel model, we will create a simple static main method that will contain all the necessary code. Of course, if you are building a production-ready application you should pay more attention to OOP concepts and SOLID principles. Java import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider; import software.amazon.awssdk.core.document.Document; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.bedrockruntime.*; import software.amazon.awssdk.services.bedrockruntime.model.*; import java.util.concurrent.CompletableFuture; public class NovaReelDemo { public static void main(String[] args) { BedrockRuntimeAsyncClient bedrockClient = BedrockRuntimeAsyncClient.builder() .region(Region.US_EAST_1) .credentialsProvider(ProfileCredentialsProvider.create()) .build(); String s3Bucket = "s3://{your_bucket_name}"; String prompt = "Fluffy cat relaxing on the beach next to the ocean under the blue sky. There should be an airplane flying in the sky"; //Create request as an instance of Document class Document novaRequest = prepareDocument(prompt); // Create request StartAsyncInvokeRequest request = StartAsyncInvokeRequest.builder() .modelId("amazon.nova-reel-v1:0") .modelInput(novaRequest) .outputDataConfig(AsyncInvokeOutputDataConfig.builder() .s3OutputDataConfig(AsyncInvokeS3OutputDataConfig.builder().s3Uri(s3Bucket).build()) .build()) .build(); try { CompletableFuture<StartAsyncInvokeResponse> startAsyncInvokeResponseCompletableFuture = bedrockClient.startAsyncInvoke(request); StartAsyncInvokeResponse startAsyncInvokeResponse = startAsyncInvokeResponseCompletableFuture.get(); System.out.println("invocation ARN: " +startAsyncInvokeResponse.invocationArn()); } catch (Exception e) { System.out.println(e); } finally { bedrockClient.close(); } } private static Document prepareDocument(String prompt) { Document textToVideoParams = Document.mapBuilder() .putString("text", prompt) .build(); Document videoGenerationConfig = Document.mapBuilder() .putNumber("durationSeconds", 6) .putNumber("fps", 24) .putString("dimension", "1280x720") .build(); Document novaRequest = Document.mapBuilder() .putString("taskType", "TEXT_VIDEO") .putDocument("textToVideoParams", textToVideoParams) .putDocument("videoGenerationConfig", videoGenerationConfig) .build(); return novaRequest; } } Let’s go block by block: We are creating BedrockRuntimeAsyncClient, providing us-east-1 as a region because it is the only region where Amazon Nova models are available at the moment. Also, using credentialsProvider(ProfileCredentialsProvider.create()), we are setting up authentication using the AWS CLI config we set up in step 1.String s3Bucket = "s3://{your_bucket_name}" – Put the bucket name you created in step 3. This is the s3 bucket where our generated video will be stored. String prompt is the actual prompt that will be used for video generation.Document novaRequest = prepareDocument(prompt) – To pass all the necessary parameters, we need to create an instance of the document class. Below you can find JSON, which might be used if we try to execute the same command using AWS CLI. When using Java, we need to wrap it in the instance of a document. JSON { "taskType": "TEXT_VIDEO", "textToVideoParams": { "text": "Fluffy cat relaxing on the beach next to the ocean under the blue sky. There should be an airplane flying in the sky" }, "videoGenerationConfig": { "durationSeconds": 6, "fps": 24, "dimension": "1280x720" } } Let’s deep dive into the configs we provided: taskType TEXT_VIDEO – Should be constant for us as we are planning only to create a video from prompts.text (required) – A text prompt to generate the video; must be 1-512 characters in length.durationSeconds (required) – Duration of the output video. 6 is the only supported value at the moment of writing this article.fps (required) – Frame rate of the output video. 24 is the only supported value at the moment of writing this article.dimension (required) – Width and height of the output video. "1280x720" is the only supported value currently. You may find more information on video generation input parameters in official AWS docs. StartAsyncInvokeRequest – Request object, which includes all the info we previously configured, including request and S3 bucket where the result video will be created and stored. Also, this request includes modelId. At the moment, amazon.nova-reel-v1:0 is the only available model for generating video in Amazon Bedrock.Request invocation using bedrockClient.startAsyncInvoke(request) returns an instance of CompletableFuture. When the CompletableFuture is completed, we can fetch the Invocation ARN to track the status of the video generation job. Step 6: Run Our Application To run our application, we need to simply execute its main method. This is a standard Java application with a public static void main(String[] args) entry point, meaning it can be launched directly from the command line or an IDE. No additional setup is required — just compile and run it as a typical Java program. In case of successful execution, you should see the invocation ARN printed in the console. You may track the status of the invocation leveraging the getAsyncInvoke command. As soon as you trigger a new invocation, you may check the S3 bucket we created in step 3. As previously mentioned, a new folder should have been created for the invocation. If you dive into this folder, you will see a manifest.json file. It usually takes up to 5 to 6 minutes for the video to be generated. Try to refresh the page after this period of time, and you will be able to see the result. Congratulations! Conclusion In this article, we explored how to integrate Java with Amazon Bedrock and the Amazon Nova Reel model to generate videos from text prompts. We walked through the necessary steps, including setting up AWS credentials, configuring the Bedrock API, creating an S3 bucket for storing results, and writing the Java code to invoke the model. Since Amazon Nova is a relatively new foundational model, Java-specific examples are scarce, making this guide a valuable resource for developers looking to work with AWS Bedrock in Java. While this was a simple implementation, it laid the groundwork for more advanced use cases, such as integrating video generation into a larger application or optimizing the process for production-ready environments. As AWS continues to expand its AI capabilities, we can expect further enhancements and new features for Amazon Nova. Stay tuned for updates, and happy coding!

By Danil Temnikov CORE

With KRaft for Kafka ZooKeeper is no longer needed. KRaft is a protocol to select a leader among several server instances. That makes the Kafka setup much easier. The new configuration is shown on the example of the MovieManager project. Using Kafka for Development For development, Kafka can be used with a simple Docker setup. That can be found in the runKafka.sh script: Shell #!/bin/sh # network config for KRaft docker network create app-tier --driver bridge # Kafka with KRaft docker run -d \ -p 9092:9092 \ --name kafka-server \ --hostname kafka-server \ --network app-tier \ -e KAFKA_CFG_NODE_ID=0 \ -e KAFKA_CFG_PROCESS_ROLES=controller,broker \ -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \ -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP= CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-server:9093 \ -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \ bitnami/kafka:latest # Start Kafka with KRaft docker start kafka-server First, the Docker bridge network app-tier is created to enable the KRaft communication of the Kafka instances among each other. Then, the Kafka instances are started with the docker run command. The port 9092 needs to be exported. After the run command has been executed, the kafka-server configuration can be started and stopped with docker commands. To run a Spring Boot application with Kafka in development, a kafka profile can be used. That enables application configurations with and without Kafka. An example configuration can look like the application-kafka.properties file: Properties files kafka.server.name=${KAFKA_SERVER_NAME:kafka-server} spring.kafka.bootstrap-servers=localhost:9092 spring.kafka.producer.compression-type=gzip spring.kafka.producer.transaction-id-prefix: tx- spring.kafka.producer.key-serializer= org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer= org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.enable.idempotence=true spring.kafka.consumer.group-id=group_id spring.kafka.consumer.auto-offset-reset=earliest spring.kafka.consumer.key-deserializer= org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.value-deserializer= org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.enable-auto-commit=false spring.kafka.consumer.isolation-level=read_committed spring.kafka.consumer.transaction-id-prefix: tx- The important lines are the first two. In the first line, the kafka.server.name is set to kafka-server, and in the second line, the spring.kafka.bootstrap-servers are set to localhost:9092. That instructs the application to connect to the dockerized Kafka instances on localhost. To connect to Kafka locally, the DefaultHostResolver has to be patched: Java public class DefaultHostResolver implements HostResolver { public static volatile String IP_ADDRESS = ""; public static volatile String KAFKA_SERVER_NAME = ""; public static volatile String KAFKA_SERVICE_NAME = ""; @Override public InetAddress[] resolve(String host) throws UnknownHostException { if(host.startsWith(KAFKA_SERVER_NAME) && !IP_ADDRESS.isBlank()) { InetAddress[] addressArr = new InetAddress[1]; addressArr[0] = InetAddress.getByAddress(host, InetAddress.getByName(IP_ADDRESS).getAddress()); return addressArr; } else if(host.startsWith(KAFKA_SERVER_NAME) && !KAFKA_SERVICE_NAME.isBlank()) { host = KAFKA_SERVICE_NAME; } return InetAddress.getAllByName(host); } } The DefaultHostResolver handles the name resolution of the Kafka server hostname. It checks if the KAFKA_SERVER_NAME is set and then sets the IP_ADDRESS for it. That is needed because the local name resolution of kafka-server does not work(unless you put it in the hosts file). Using Kafka in a Kubernetes Deployment For a Kubernetes deployment, an updated configuration is needed. The values.yaml changes look like this: YAML ... kafkaServiceName: kafkaservice ... secret: nameApp: app-env-secret nameDb: db-env-secret nameKafka: kafka-env-secret envKafka: normal: KAFKA_CFG_NODE_ID: 0 KAFKA_CFG_PROCESS_ROLES: controller,broker KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafkaservice:9093 KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER Because ZooKeeper is no longer needed, all of that configuration has been removed. The Kafka configuration is updated to the values KRaft needs. In the template, the ZooKeeper configuration has been removed. The Kafka deployment configuration has not changed because only the parameters change. The parameters are provided by the values.yaml with the helpers.tpl script. The Kafka service configuration template needs to change to support the KRaft communication between the Kafka instances for leader selection: YAML apiVersion: v1 kind: Service metadata: name: {{ .Values.kafkaServiceName } labels: app: {{ .Values.kafkaServiceName } spec: ports: - name: tcp-client port: 9092 protocol: TCP - name: tcp-interbroker port: 9093 protocol: TCP targetPort: 9093 selector: app: {{ .Values.kafkaName } This is a normal service configuration that opens the port 9092 internally and works for the Kafka deployment. The tcp-interbroker configuration is for the KRaft leader selection. It opens the port '9093' internally and provides the 'targetPort' to enable sending requests among each other. The application can now be run with the profiles prod and kafka and will start with the application-prod-kafka.properties configuration: Properties files kafka.server.name=${KAFKA_SERVER_NAME:kafkaapp} spring.kafka.bootstrap-servers=${KAFKA_SERVICE_NAME}:9092 spring.kafka.producer.compression-type=gzip spring.kafka.producer.transaction-id-prefix: tx- spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.consumer.group-id=group_id spring.kafka.consumer.auto-offset-reset=earliest spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.enable-auto-commit=false spring.kafka.consumer.isolation-level=read_committed spring.kafka.consumer.transaction-id-prefix: tx- The deployment configuration is very similar to the development configuration. The main difference is in the first two lines. The application will not start if the environment variable KAFKA_SERVICE_NAME is not set. That would be an error in the deployment configuration of the MovieManager application that has to be fixed. Conclusion With KRaft, Kafka no longer needs ZooKeeper, making the configuration simpler. For development, two Docker commands and a simple Spring Boot configuration are enough to start Kafka instances to develop against. For deployment in Kubernetes, the configuration of the Docker commands for the development setup can be used to create Kafka instances, and a second port configuration is needed for KRaft. The time of Kafka being more difficult than other messaging solutions is over. It is now easy to develop and easy to deploy. Kafka should now be used for all the use cases where it fits the requirements.

The Initial Hurdle In NLP classification tasks, especially those involving text descriptions, it's common to encounter two significant hurdles: large model sizes and imbalanced datasets. A large model can be difficult to deploy and manage, while class imbalance can lead to poor predictive performance, particularly for minority classes. Imagine you are building a system to automatically categorize product descriptions into various product categories. The project began with a dataset of close to 40,000 records, where each record contained a short product title and a longer product description, along with its corresponding category. The initial Random Forest model, while achieving a decent accuracy of around 70%, ballooned to a whopping 11 GB size. Further experimentation revealed that tuning the model parameters caused a drastic drop in accuracy (to 14%), rendering the model practically useless. This situation highlights the core challenges: Large model size: An 11 GB model is unwieldy and resource-intensive.Class imbalance: Uneven distribution of product categories makes it hard for the model to learn effectively.Overfitting: Parameter tuning leads to a significant drop in accuracy, indicating that the model is overfitting the training data. The Solutions Explored To address these issues, the focus was on reducing the model size and improving its accuracy. Here's a rundown of the techniques tried, the challenges faced, and what ultimately worked: 1. Addressing Class Imbalance Resampling Techniques (SMOTE and ADASYN) Oversampling the minority classes using SMOTE (Synthetic Minority Oversampling Technique) and ADASYN (Adaptive Synthetic Sampling Approach) was explored. The goal was to artificially increase the number of samples in the less frequent categories. However, several issues were encountered: ValueError: Expected n_neighbors <= n_samples_fit. This error occurred when the number of nearest neighbors required by SMOTE/ADASYN exceeded the number of available samples in a minority class.Low accuracy. Even when the error was avoided, the resulting models often had lower accuracy than the baseline, suggesting that the synthetic samples were introducing noise or not effectively improving the model's ability to generalize. The oversampled data sometimes led to the model "memorizing" the synthetic examples, leading to poor performance on unseen data. Here is a sample code with SMOTE: Python from imblearn.over_sampling import SMOTE from sklearn.model_selection import train_test_split # Assuming X is your features and y is your labels X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y) # Split data smote = SMOTE(random_state=42) # Adjust k_neighbors X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train) print(f"Shape of X_train before SMOTE: {X_train.shape}") print(f"Shape of X_train after SMOTE: {X_train_resampled.shape}") Here are some key considerations for the SMOTE method: Apply resampling only to the training data (not the testing set) to prevent data leakage.Carefully tune the k_neighbors parameter in SMOTE/ADASYN. A value that is too high can lead to errors, while a value that is too low might not generate effective synthetic samples.Use stratified sampling during train/test split to maintain class proportions. This ensures that the class distribution in the training and testing sets is representative of the overall dataset. Cost-Sensitive Learning Higher misclassification costs were assigned to minority classes by using the class_weight='balanced' or class_weight='balanced_subsample' parameter in the Random Forest model. This tells the model to penalize misclassifying minority class examples more heavily. Python pythonfrom sklearn.ensemble import RandomForestClassifier rf_model = RandomForestClassifier( n_estimators=100, # Or your chosen number of trees class_weight='balanced_subsample', # Or 'balanced' random_state=42, n_jobs=-1 # Use all available cores ) Threshold Adjustment Adjusting the classification threshold to favor the minority class was considered. This involves analyzing the predicted probabilities and choosing a threshold that works well for your specific problem. However, this requires careful evaluation on a validation set to avoid overfitting. 2. Improving Model Accuracy Feature Engineering Given the text data, more informative features were created: TF-IDF (Term Frequency-Inverse Document Frequency) with N-grams: This technique captures the importance of words in a document relative to the entire corpus, while N-grams capture phrases and word order information.Word embeddings (Word2Vec, GloVe, FastText): These represent words as dense vectors, capturing semantic relationships between words.Length of description: The length of the text description itself could be a useful feature.Number of keywords: The occurrence of specific keywords related to different product categories was counted. Model Complexity and Regularization The complexity of the Random Forest model was reduced by: Reducing n_estimators: The number of trees in the Random Forest is a major contributor to its size.Limiting max_depth: Restricting the maximum depth of the trees prevents them from becoming overly complex.Increasing min_samples_split and min_samples_leaf: These parameters control the minimum number of samples required to split a node and to be at a leaf node, respectively. Increasing them forces the trees to be less specific. Model Selection Simpler models like Multinomial Naive Bayes, Logistic Regression, and Linear SVM were considered, as Random Forest can be prone to overfitting. Gradient Boosting Machines (GBM) and XGBoost were also tried. Hyperparameter Optimization Cross-validation was used to evaluate model performance and tune hyperparameters. Randomized search and Bayesian optimization were employed to efficiently explore the hyperparameter space. Python from sklearn.model_selection import train_test_split, RandomizedSearchCV from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline pipeline = Pipeline([ ('tfidf', TfidfVectorizer(stop_words='english', max_features=5000)), # Adjust max_features ('classifier', LogisticRegression(solver='liblinear', multi_class='ovr', random_state=42, class_weight='balanced')) # Use class_weight ]) param_grid = { 'tfidf__ngram_range': [(1, 1), (1, 2)], 'tfidf__use_idf': [True, False], 'classifier__C': [0.1, 1, 10], 'classifier__penalty': ['l1', 'l2'] } randomized_search = RandomizedSearchCV(pipeline, param_distributions=param_grid, n_iter=10, cv=3, scoring='accuracy', random_state=42, n_jobs=-1) randomized_search.fit(X_train, y_train) 3. Reducing Model Size Reducing Vocabulary Size The vocabulary size of the text vectorizer was limited using max_features in TfidfVectorizer. Quantization Model quantization techniques were explored to reduce model size, but this required using a different framework like TensorFlow Lite or ONNX Runtime, which was too much work for such a small task. What Worked Ultimately Ultimately, a significantly smaller model size was achieved while maintaining acceptable accuracy by: Removing infrequent classes (with a higher threshold). Removing categories with fewer than X occurrences significantly improved model performance. This created a more stable and manageable dataset, despite reducing the overall size. Combining textual information. The product title and product description were combined into a single text field. This allowed the model to learn relationships between the concise title and the more detailed description.TF-IDF with limited vocabulary. Using TF-IDF with a limited vocabulary size (max_features=5000) struck a good balance between accuracy and model size. This prevented the model from overfitting to rare or irrelevant words.Removing samples with less than X (this will depend on your data) occurrences.Combining the textual information from the product title and product description into a single feature.Using TF-IDF with max_features=5000. Here's a sample of the data: product_titleproduct_descriptionproduct_categoryLaptopRAM size foldable screen, touchElectronics & Computers Nonstick potScratched after a month, but maybe that’s my fault for using a metal spoonKitchen & CookwareCeiling FanRemote works from my bed—life is good.Home & AppliancesLaptop BagWater-resistant? Yes. Waterproof? Nope.Accessories & BagsBookShelfAssembly took me an hour and my sanity, but it looks great!Furniture & Storage Lessons Learned This experience underscored several key lessons: Data cleaning is paramount. Thoroughly examine your data, clean it, and handle missing values appropriately. Inconsistent data can significantly hinder model performance.Class imbalance requires careful consideration. While resampling techniques can be helpful, they're not always the best solution. Sometimes, a more aggressive filtering approach can be more effective.Feature engineering matters. Combining existing features or creating new ones can significantly improve model accuracy.Model complexity is a trade-off. Simpler models are often more robust and easier to manage than complex ones.Experimentation is essential. Don't be afraid to try different approaches and evaluate their performance. By systematically addressing these issues, you can build NLP classification models that are both accurate and manageable.

For the last five years, I’ve had the quote “Everything begins with an idea” on the wall of my office. My wife found this product on Etsy shortly after I started developing an API collection for a fitness application. I love this statement because it captures the passion that consumes me during the creation stages of a new project. This is still my favorite aspect of being an engineer, even three decades into my career. What I’ve learned during this time is that an idea only matters if someone has the opportunity to experience it. If an idea takes too long to become a reality, you end up with a missed opportunity as someone else beats you to the punch. This is why startups are always racing to get their ideas to market as quickly as possible. Let’s walk through how we can make an idea a reality... quickly. Assumptions For this article, we’ll keep things simple. We’ll use Java 17 and Spring Boot 3 to create a RESTful API. In this example, we’ll use Gradle for our build automation. While the service idea we plan to take to market would normally use a persistence layer, we’ll set that aside for this example and statically define our data within a repository class. We won’t worry about adding any security for this example, simply allowing anonymous access for this proof of concept. The Motivational Quotes API Let’s assume our idea is a motivational quotes API. To make sure we are racing as fast as possible, I asked ChatGPT to create an OpenAPI spec for me. Within seconds, ChatGPT provided the response: Here’s the OpenAPI specification in YAML that ChatGPT generated: YAML openapi: 3.0.0 info: title: Motivational Quotes API description: An API that provides motivational quotes. version: 1.0.0 servers: - url: https://api.example.com description: Production server paths: /quotes: get: summary: Get all motivational quotes operationId: getAllQuotes responses: '200': description: A list of motivational quotes content: application/json: schema: type: array items: $ref: '#/components/schemas/Quote' /quotes/random: get: summary: Get a random motivational quote operationId: getRandomQuote responses: '200': description: A random motivational quote content: application/json: schema: $ref: '#/components/schemas/Quote' /quotes/{id}: get: summary: Get a motivational quote by ID operationId: getQuoteById parameters: - name: id in: path required: true schema: type: integer responses: '200': description: A motivational quote content: application/json: schema: $ref: '#/components/schemas/Quote' '404': description: Quote not found components: schemas: Quote: type: object required: - id - quote properties: id: type: integer quote: type: string I only needed to make one manual update — making sure the id and quote properties were required for the Quote schema. And that’s only because I forgot to mention this constraint to ChatGPT in my original prompt. With that, we’re ready to develop the new service using an API-first approach. Building the Spring Boot Service Using API-First For this example, I’ll use the Spring Boot CLI to create a new project. Here’s how you can install the CLI using Homebrew: Shell $ brew tap spring-io/tap $ brew install spring-boot Create a New Spring Boot Service We’ll call the project quotes, creating it with the following command: Shell $ spring init --dependencies=web quotes Let’s examine the contents of the quotes folder: Shell $ cd quotes && ls -la total 72 drwxr-xr-x@ 11 jvester 352 Mar 1 10:57 . drwxrwxrwx@ 90 jvester 2880 Mar 1 10:57 .. -rw-r--r--@ 1 jvester 54 Mar 1 10:57 .gitattributes -rw-r--r--@ 1 jvester 444 Mar 1 10:57 .gitignore -rw-r--r--@ 1 jvester 960 Mar 1 10:57 HELP.md -rw-r--r--@ 1 jvester 545 Mar 1 10:57 build.gradle drwxr-xr-x@ 3 jvester 96 Mar 1 10:57 gradle -rwxr-xr-x@ 1 jvester 8762 Mar 1 10:57 gradlew -rw-r--r--@ 1 jvester 2966 Mar 1 10:57 gradlew.bat -rw-r--r--@ 1 jvester 28 Mar 1 10:57 settings.gradle drwxr-xr-x@ 4 jvester 128 Mar 1 10:57 src Next, we edit the build.gradle file as shown below to adopt the API-first approach. Groovy plugins { id 'java' id 'org.springframework.boot' version '3.4.3' id 'io.spring.dependency-management' version '1.1.7' id 'org.openapi.generator' version '7.12.0' } openApiGenerate { generatorName = "spring" inputSpec = "$rootDir/src/main/resources/static/openapi.yaml" outputDir = "$buildDir/generated" apiPackage = "com.example.api" modelPackage = "com.example.model" configOptions = [ dateLibrary: "java8", interfaceOnly: "true", useSpringBoot3: "true", useBeanValidation: "true", skipDefaultInterface: "true" ] } group = 'com.example' version = '0.0.1-SNAPSHOT' java { toolchain { languageVersion = JavaLanguageVersion.of(17) } } repositories { mavenCentral() } dependencies { implementation 'org.springframework.boot:spring-boot-starter-web' implementation 'org.springframework.boot:spring-boot-starter-validation' implementation 'org.openapitools:jackson-databind-nullable:0.2.6' implementation 'io.swagger.core.v3:swagger-annotations:2.2.20' annotationProcessor 'org.projectlombok:lombok' testImplementation 'org.springframework.boot:spring-boot-starter-test' testRuntimeOnly 'org.junit.platform:junit-platform-launcher' } sourceSets { main { java { srcDirs += "$buildDir/generated/src/main/java" } } } compileJava.dependsOn tasks.openApiGenerate tasks.named('test') { useJUnitPlatform() } Finally, we place the generated OpenAPI specification into the resources/static folder as openapi.yaml. Generate the API and Model Objects After opening the project in IntelliJ, I executed the following command to build the API stubs and model objects. Shell ./gradlew clean build Now, we can see the api and model objects created from our OpenAPI specification. Here’s the QuotesAPI.java file: Add the Business Logic With the base service ready and already adhering to our OpenAPI contract, we will start adding some business logic to the service. First, we create a QuotesRepository class, which returns the data for our service. As noted above, this would normally be stored in some dedicated persistence layer. For this example, hard-coding five quotes’ worth of data works just fine, and it keeps us focused. Java @Repository public class QuotesRepository { public static final List<Quote> QUOTES = List.of( new Quote() .id(1) .quote("The greatest glory in living lies not in never falling, but in rising every time we fall."), new Quote() .id(2) .quote("The way to get started is to quit talking and begin doing."), new Quote() .id(3) .quote("Your time is limited, so don't waste it living someone else's life."), new Quote() .id(4) .quote("If life were predictable it would cease to be life, and be without flavor."), new Quote() .id(5) .quote("If you set your goals ridiculously high and it's a failure, you will fail above everyone else's success.") ); public List<Quote> getAllQuotes() { return QUOTES; } public Optional<Quote> getQuoteById(Integer id) { return Optional.ofNullable(QUOTES.stream().filter(quote -> quote.getId().equals(id)).findFirst().orElse(null)); } } Next, we create a QuotesService which will interact with the QuotesRepository. Taking this approach will keep the data separate from the business logic. Java @RequiredArgsConstructor @Service public class QuotesService { private final QuotesRepository quotesRepository; public List<Quote> getAllQuotes() { return quotesRepository.getAllQuotes(); } public Optional<Quote> getQuoteById(Integer id) { return quotesRepository.getQuoteById(id); } public Quote getRandomQuote() { List<Quote> quotes = quotesRepository.getAllQuotes(); return quotes.get(ThreadLocalRandom.current().nextInt(quotes.size())); } } Finally, we just need to implement the QuotesApi generated from our API-first approach: Java @Controller @RequiredArgsConstructor public class QuotesController implements QuotesApi { private final QuotesService quotesService; @Override public ResponseEntity<List<Quote>> getAllQuotes() { return new ResponseEntity<>(quotesService.getAllQuotes(), HttpStatus.OK); } @Override public ResponseEntity<Quote> getQuoteById(Integer id) { return quotesService.getQuoteById(id) .map(quote -> new ResponseEntity<>(quote, HttpStatus.OK)) .orElseGet(() -> new ResponseEntity<>(HttpStatus.NOT_FOUND)); } @Override public ResponseEntity<Quote> getRandomQuote() { return new ResponseEntity<>(quotesService.getRandomQuote(), HttpStatus.OK); } } At this point, we have a fully functional Motivational Quotes API, complete with a small collection of responses. Some Final Items Spring Boot gives us the option for a web-based Swagger Docs user interface via the springdoc-openapi-starter-webmvc-ui dependency. Groovy dependencies { ... implementation 'org.springdoc:springdoc-openapi-starter-webmvc-ui:2.8.5' ... } While the framework allows engineers to use simple annotations to describe their API, we can use our existing openapi.yaml file in the resources/static folder. We can implement this approach in the application-properties.yaml file, along with a few other minor configuration updates: YAML server: port: ${PORT:8080} spring: application: name: quotes springdoc: swagger-ui: path: /swagger-docs url: openapi.yaml Just for fun, let’s add a banner.txt file for use when the service starts. We place this file into the resources folder. Shell ${AnsiColor.BLUE} _ __ _ _ _ ___ | |_ ___ ___ / _` | | | |/ _ \| __/ _ \/ __| | (_| | |_| | (_) | || __/\__ \ \__, |\__,_|\___/ \__\___||___/ |_| ${AnsiColor.DEFAULT} :: Running Spring Boot ${AnsiColor.BLUE}${spring-boot.version}${AnsiColor.DEFAULT} :: Port #${AnsiColor.BLUE}${server.port}${AnsiColor.DEFAULT} :: Now, when we start the service locally, we can see the banner: Once started, we can validate the Swagger Docs are working by visiting the /swagger-docs endpoint. Finally, we’ll create a new Git-based repository so that we can track any future changes: Shell $ git init $ git add . $ git commit -m "Initial commit for the Motivational Quotes API" Now, let’s see how quickly we can deploy our service. Using Heroku to Finish the Journey So far, the primary focus for introducing my new idea has been creating an OpenAPI specification and writing some business logic for my service. Spring Boot handled everything else for me. When it comes to running my service, I prefer to use Heroku because it’s a great fit for Spring Boot services. I can deploy my services quickly without getting bogged down with cloud infrastructure concerns. Heroku also makes it easy to pass in configuration values for my Java-based applications. To match the Java version we’re using, we create a system.properties file in the root folder of the project. The file has one line: Shell java.runtime.version = 17 Then, I create a Procfile in the same location for customizing the deployment behavior. This file also has one line: Shell web: java -jar build/libs/quotes-0.0.1-SNAPSHOT.jar It’s time to deploy. With the Heroku CLI, I can deploy the service using a few simple commands. First, I authenticate the CLI and then create a new Heroku app. Shell $ heroku login $ heroku create Creating app... done, vast-crag-43256 https://vast-crag-43256-bb5e35ea87de.herokuapp.com/ | https://git.heroku.com/vast-crag-43256.git My Heroku app instance is named vast-crag-43256 (I could have passed in a specified name), and the service will run at https://vast-crag-43256-bb5e35ea87de.herokuapp.com/. The last thing to do is deploy the service by using a Git command to push the code to Heroku: Shell $ git push heroku master Once this command is complete, we can validate a successful deployment via the Heroku dashboard: Now, we’re ready to take our new service for a test drive! Motivational Quotes in Action With the Motivational Quotes service running on Heroku, we can validate everything is working as expected using a series of curl commands. First, let’s get a complete list of all five motivational quotes: Shell $ curl \ --location 'https://vast-crag-43256-bb5e35ea87de.herokuapp.com/quotes' JSON [ { "id":1, "quote":"The greatest glory in living lies not in never falling, but in rising every time we fall." }, { "id":2, "quote":"The way to get started is to quit talking and begin doing." }, { "id":3, "quote":"Your time is limited, so don't waste it living someone else's life." }, { "id":4, "quote":"If life were predictable it would cease to be life, and be without flavor." }, { "id":5, "quote":"If you set your goals ridiculously high and it's a failure, you will fail above everyone else's success." } ] Let’s retrieve a single motivational quote by ID: Shell $ curl \ --location 'https://vast-crag-43256-bb5e35ea87de.herokuapp.com/quotes/3' JSON { "id":3, "quote":"Your time is limited, so don't waste it living someone else's life." } Let’s get a random motivational quote: Shell $ curl --location \ 'https://vast-crag-43256-bb5e35ea87de.herokuapp.com/quotes/random' JSON { "id":5, "quote":"If you set your goals ridiculously high and it's a failure, you will fail above everyone else's success." } We can even browse the Swagger Docs, too. Conclusion Time to market can make or break any idea. This is why startups are laser-focused on delivering their innovations as quickly as possible. The longer it takes to reach the finish line, the greater the risk of a competitor arriving before you. My readers may recall my personal mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” — J. Vester In this article, we saw how Spring Boot handled everything required to implement a RESTful API. Leveraging ChatGPT, we were even able to express in human words what we wanted our service to be, and it created an OpenAPI specification for us in a matter of seconds. This allowed us to leverage an API-First approach. Once ready, we were able to deliver our idea using Heroku by issuing a few CLI commands. Spring Boot, ChatGPT, and Heroku provided the frameworks and services so that I could remain laser-focused on realizing my idea. As a result, I was able to adhere to my personal mission statement and, more importantly, deliver my idea quickly. All I had to do was focus on the business logic behind my idea—and that’s the way it should be! If you’re interested, the source code for this article can be found on GitLab. Have a really great day!

By John Vester CORE

DZone events bring together industry leaders, innovators, and peers to explore the latest trends, share insights, and tackle industry challenges. From Virtual Roundtables to Fireside Chats, our events cover a wide range of topics, each tailored to provide you, our DZone audience, with practical knowledge, meaningful discussions, and support for your professional growth. DZone Events Happening Soon Below, you’ll find upcoming events that you won't want to miss. Unpacking the 2025 Developer Experience Trends Report: Insights, Gaps, and Putting it into Action Date: March 19, 2025Time: 1:00 PM ET Register for Free! We’ve just seen the 2025 Developer Experience Trends Report from DZone, and while it shines a light on important themes like platform engineering, developer advocacy, and productivity metrics, there are some key gaps that deserve attention. Join Cortex Co-founders Anish Dhar and Ganesh Datta for a special webinar, hosted in partnership with DZone, where they’ll dive into what the report gets right—and challenge the assumptions shaping the DevEx conversation. Their take? Developer experience is grounded in clear ownership. Without ownership clarity, teams face accountability challenges, cognitive overload, and inconsistent standards, ultimately hampering productivity. Don’t miss this deep dive into the trends shaping your team’s future. Accelerating Software Delivery: Unifying Application and Database Changes in Modern CI/CD Date: March 25, 2025Time: 1:00 PM ET Register for Free! Want to speed up your software delivery? It’s time to unify your application and database changes. Join us for Accelerating Software Delivery: Unifying Application and Database Changes in Modern CI/CD, where we’ll teach you how to seamlessly integrate database updates into your CI/CD pipeline. Petabyte Scale, Gigabyte Costs: Mezmo’s ElasticSearch to Quickwit Evolution Date: March 27, 2025Time: 1:00 PM ET Register for Free! For Mezmo, scaling their infrastructure meant facing significant challenges with ElasticSearch. That's when they made the decision to transition to Quickwit, an open-source, cloud-native search engine designed to handle large-scale data efficiently. This is a must-attend session for anyone looking for insights on improving search platform scalability and managing data growth. Best Practices for Building Secure Data Pipelines with Apache Airflow® Date: April 15, 2025Time: 1:00 PM ET Register for Free! Security is a critical but often overlooked aspect of data pipelines. Effective security controls help teams protect sensitive data, meet compliance requirements with confidence, and ensure smooth, secure operations. Managing credentials, enforcing access controls, and ensuring data integrity across systems can become overwhelming—especially while trying to keep Airflow environments up–to-date and operations running smoothly. Whether you're working to improve access management, protect sensitive data, or build more resilient pipelines, this webinar will provide the knowledge and best practices to enhance security in Apache Airflow. Generative AI: The Democratization of Intelligent Systemsive Date: April 16, 2025Time: 1:00 PM ET Register for Free! Join DZone, alongside industry experts from Cisco and Vertesia, for an exclusive virtual roundtable exploring the latest trends in GenAI. This discussion will dive into key insights from DZone's 2025 Generative AI Trend Report, focusing on advancements in GenAI models and algorithms, their impact on code generation, and the evolving role of AI in software development. We’ll examine AI adoption maturity, intelligent search capabilities, and how organizations can optimize their AI strategies for 2025 and beyond. What's Next? DZone has more in store! Stay tuned for announcements about upcoming Webinars, Virtual Roundtables, Fireside Chats, and other developer-focused events. Whether you’re looking to sharpen your skills, explore new tools, or connect with industry leaders, there’s always something exciting on the horizon. Don’t miss out — save this article and check back often for updates!

TL; DR: Vibe Coding Vibe coding — using natural language to generate code through AI — represents a significant evolution in software development. It accelerates feedback cycles and democratizes programming but raises concerns about maintainability, security, and technical debt. Learn why success likely requires a balanced approach: using vibe coding for rapid prototyping while maintaining rigorous standards for production code, with developers evolving from writers to architects and reviewers or auditors. Definition and Origins of Vibe Coding In a way, vibe coding represents a fundamental shift in creating software. Rather than typing out syntax-perfect code, developers using the vibe coding approach express their intent in natural language, letting AI tools handle the translation into functional code. This approach emerged from the rapidly evolving capabilities of large language models (LLMs) and coding assistants. The term gained significant traction following Andrej Karpathy’s observations about the changing nature of programming, capturing the paradigm shift as developers began leveraging AI to convert natural language descriptions into executable code. As these AI coding assistants matured, Karpathy’s insight evolved further to the point where he noted that with modern AI tools, “it’s not really coding — I just see stuff, say stuff, run stuff, and copy-paste stuff, and it mostly works.” This statement encapsulates the essence of vibe coding: describing what you want the software to do and trusting AI to implement it correctly, often with minimal human intervention in the actual code generation process. Agile/Product Management Perspective Vibe Coding and Alignment With Agile Principles From an Agile perspective, vibe coding presents intriguing possibilities. The core Agile principle of “responding to change over following a plan” perfectly aligns with vibe coding’s flexibility. When product requirements shift, describing the new functionality in natural language and having AI quickly generate updated code could theoretically accelerate the adaptation process. Vibe coding also resonates with the “inspect and adapt” cycle central to Scrum and other Agile frameworks. Teams can quickly generate code, inspect the results, learn from feedback, and iterate rapidly — potentially compressing what might have been days of conventional development into hours. Accelerating the Build-Measure-Learn Cycle Vibe coding could turbocharge the “build, measure, learn” cycle for product managers and entrepreneurs practicing Lean. Creating functional prototypes to test with users no longer requires coding knowledge or a significant investment. A product manager could describe a feature in natural language, have AI generate a working prototype, and put it in front of users within the same day. This acceleration of the feedback loop represents perhaps the strongest argument for vibe coding’s Agile credentials. When gathering user feedback is the primary goal, functional demonstrations that produce meaningful responses become more valuable. Moving Fast Without Breaking (Too Many) Things Vibe coding could enable teams to move at remarkable speeds, but questions remain about reliability. While AI-generated code often “mostly works,” that qualifier becomes problematic in production environments where “mostly” isn’t good enough. However, the speed advantages may outweigh the quality concerns for early-stage validation and prototyping. Innovative product teams might adopt a hybrid approach: using vibe coding for rapid prototyping and initial validation, then refactoring critical components with traditional coding practices once the direction is validated. This balanced approach could capture speed benefits while mitigating risks. Technical Perspective Quality and Maintainability Concerns Professional developers raise legitimate concerns about code quality with vibe coding approaches. AI-generated code, while functional, often lacks the thoughtful architecture and dedication to technical excellence that experienced developers bring to projects. As Toby Devonshire warns: “Vibe coding is all fun and games until you have to vibe debug.” The concern is valid: prototypes may morph into production systems riddled with technical debt without disciplined testing and documentation. This effect can result in inadequate implementations, unnecessary dependencies, or brittle code that works for the specific use case but breaks when conditions change. For example, Jo Bergum reports: “My experience with vibe coding is that it’s fantastic for MVP but more frustrating for rewrites in larger code bases. When Claude starts to fake implementations to make tests pass, or solve dependency issues by implementing a mock, it feels like there is still a few more months“. Therefore, perhaps, maintainability presents the greatest technical challenge. When developers need to modify AI-generated code months later, they may struggle to understand the implementation choices. Without clear documentation explaining why certain approaches were taken, maintenance becomes significantly more difficult. Security Vulnerabilities Security represents another major concern. AI coding assistants primarily optimize for functionality rather than security, potentially introducing vulnerabilities that human developers would quickly identify. While some AI tools incorporate security checks, they rarely match the comprehensive risk analysis that security-focused developers perform. These concerns become especially relevant for applications handling sensitive data or financial transactions. Vibe coding might be inappropriate for these use cases without extensive human review and enhancement. Technical Debt Acceleration Perhaps most concerning from a technical perspective is the potential for vibe coding to accelerate technical debt accumulation. Traditional development approaches often involve careful consideration of architecture to minimize future rework. Vibe coding, emphasizing immediate functionality over long-term sustainability, could create significant future costs. The process resembles taking out a high-interest loan against your codebase’s future. The immediate productivity gains feel valuable, but eventually, the technical debt payments come due in the form of maintenance challenges, refactoring needs, and scaling difficulties. Entrepreneurial and Economic Perspective Democratizing Software Creation One of vibe coding’s most promising aspects is its potential to democratize software creation. Entrepreneurs with valuable domain knowledge but limited technical skills can now transform their ideas into working software without first learning to code or hiring expensive development teams. This accessibility could unleash innovation from previously excluded groups. Domain experts, for example, in healthcare, education, agriculture, and countless other fields, might create specialized tools that address needs that traditional software companies have overlooked. Lowering Barriers to Entry The economic implications of lowering barriers to entry in software creation are significant. Startups could launch with smaller initial investments, extending their runway and increasing their chances of finding product-market fit before funds run out. This advantage could lead to more diverse experimentation in the software ecosystem. For established companies, vibe coding might enable innovation from departments traditionally separate from software development. Marketing teams could prototype customer-facing tools, operations staff could build process automation, and sales representatives could create custom demos without depending on a development team’s bandwidth. (Please note that this perspective might introduce problems at a different level, from strategic alignment to selling not features but products that do not yet exist.) Vibe Coding and Developer Job Security Some critics argue that resistance to vibe coding stems partially from developers’ concerns about job security. If non-technical team members can create functional software independently, what does this mean for professional developers? This perspective oversimplifies the situation. While vibe coding might reduce demand for basic implementation work, it likely increases the value of developers who can architect complex systems, optimize performance, ensure security, and maintain large codebases over time. The role may evolve, but professional development expertise remains essential. Balanced View and Possible Future Finding the Appropriate Applications for Vibe Coding A balanced assessment suggests vibe coding has legitimate applications within an Agile context, particularly for: Rapid prototyping and concept validation,Internal tools with limited security requirements,Projects where the speed of iteration outweighs long-term maintainability,Environments where technical resources are limited. Conversely, vibe coding appears less suitable for: Systems with stringent security requirements,Applications requiring optimal performance,Long-lived products expecting significant evolution,Projects with regulatory compliance requirements. Mitigation Strategies Therefore, it seems beneficial that teams integrating vibe coding with Agile workflows adopt hybrid approaches: Phased integration. Start with non-critical tasks like utility functions, then expand to feature development,AI-augmented reviews. Tools like Snyk now offer workshops to audit AI-generated code for vulnerabilities,Guardrails. Enforcing coding standards via prompts (e.g., “Refactor for readability”) reduces technical debt. The Evolving Role of Professional Developers As vibe coding capabilities mature, the role of professional developers will likely transform rather than diminish. Developers may shift from writing every line of code to: Defining and maintaining architectural standards,Auditing and refining AI-generated code,Creating guardrails and frameworks within which AI can safely operate,Focusing on complex problems that remain challenging for AI. This evolution mirrors historical patterns in software development. Just as high-level languages and frameworks abstracted away machine code and memory management, vibe coding may abstract away routine implementation details, allowing developers to focus on higher-level concerns. Vibe Coding’s Integration with Agile Practices Forward-thinking Agile teams might integrate vibe coding into existing workflows rather than replacing traditional development entirely. For example: Using vibe coding during initial sprints focused on concept validation,Letting Product Owners or designers create initial implementations for refinement by developers,Employing AI pair programming where developers guide AI code generation with human expertise. This integration acknowledges both vibe coding’s strengths and limitations, using each approach where it delivers the most value. Conclusion Vibe coding represents neither a complete revolution nor a mere fad but rather an evolution in how software gets created. It aligns with Agile’s emphasis on rapid iteration and user feedback while challenging traditional code quality and maintainability notions. Organizations willing to thoughtfully integrate vibe coding into their development workflows — recognizing where it adds value and where traditional approaches remain superior — stand to gain significant advantages in speed and accessibility. Those who dismiss it entirely risk missing opportunities for innovation, while those who uncritically embrace vibe coding may create unsustainable technical debt. As with most technological advances, the most successful approach involves pragmatic adoption guided by clear principles rather than wholesale acceptance or rejection. Vibe coding is unlikely to replace traditional development, but it has already begun to transform it in ways that will reshape software creation irreversibly. What are you using vibe coding for? Please share with us in the comments.

By Stefan Wolpers CORE

Hi community! This is my third article in a series of introductions to Spring AI. You may find the first two on the link below: Using Spring AI to Generate Images With OpenAI's DALL-E 3Exploring Embeddings API with Java and Spring AI In this article, I’ll skip the explanation of some basic Spring concepts like bean management, starters, etc., as the main goal of this article is to discover Spring AI capabilities. For the same reason, I’ll not create detailed instructions on generating the AWS Credentials. In case you don’t have one, follow the links in Step 0, that should give you enough context on how to create one. Also, please be aware that executing code from this application may cost you some money for running AWS Bedrock. The code I will share in this article is also available in the GitHub repo. Introduction Today, we are going to implement a simple proxy application that will forward our text prompt to a foundational model managed by AWS Bedrock and return a response. This will allow us to make our code more flexible and configurable. But before we start, let me give you a little introduction to Amazon Bedrock. Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Luma, Meta, Mistral AI, poolside (coming soon), Stability AI, and Amazon through a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI. The single-API access of Amazon Bedrock, regardless of the models you choose, gives you the flexibility to use different FMs and upgrade to the latest model versions with minimal code changes. You may find more information about Amazon Bedrock and its capabilities on the service website. Now, let’s start implementing! Step 0. Generate AWS Keys and Choose the Foundational Model to Use If you don’t have an active AWS access key, do the following steps (copy-pasted from this SOF thread): Plain Text Go to: http://aws.amazon.com/ Sign Up & create a new account (they'll give you the option for 1 year trial or similar) Go to your AWS account overview Account menu in the upper-right (has your name on it) sub-menu: Security Credentials After you have your keys generated you should choose and enable the foundational model in Bedrock. Go to Amazon Bedrock, and from the Model Access menu on the left, configure access to the models you are going to use. Step 1. Set Up a Project To quickly generate a project template with all necessary dependencies, one may use https://start.spring.io/. In my example, I’ll be using Java 17 and Spring Boot 3.4.1. Also, we need to include the following dependency: Amazon Bedrock Converse AI. Spring AI support for Amazon Bedrock Converse. It provides a unified interface for conversational AI models with enhanced capabilities, including function/tool calling, multimodal inputs, and streaming responsesWeb. Build web, including RESTful, applications using Spring MVC. Uses Apache Tomcat as the default embedded container. After clicking generate, open downloaded files in the IDE you are working on and validate that all necessary dependencies and properties exist in pom.xml. XML <properties> <java.version>17</java.version> <spring-ai.version>1.0.0-M5</spring-ai.version> </properties> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bedrock-converse-spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> At the moment of writing this article, Spring AI version 1.0.0-M5 has not been published to the central maven repository yet and is only available in the Spring Repository. That’s why we need to add a link to that repo in our pom.xml as well: XML <repositories> <repository> <id>spring-milestones</id> <name>Spring Milestones</name> <url>https://repo.spring.io/milestone</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> Step 2. Set Up the Configuration File As a next step, we need to configure our property file. By default, Spring uses application.yaml or application.properties file. In this example, I’m using the YAML format. You may reformat code into .properties if you feel more comfortable working with this format. Here are all the configs we need to add to the application.yaml file: YAML spring: application: name: aichatmodel ai: bedrock: aws: access-key: [YOUR AWS ACCCESS KEY] secret-key: [YOUR AWS SECRET KEY] converse: chat: options: model: amazon.titan-text-express-v1 Model: The model ID to use. You can use the supported models and model features. Model is the only required config to start our application. All the other configs are not required or have default values. As the main purpose of this article is to show the ease of Spring AI integration with AWS Bedrock, we will not go deeper into other configurations. You may find a list of all configurable properties in the Spring Boot docs. Step 3: Create a Chat Service Now, let’s create the main service of our application -> chat service. To integrate our application with the Chat Model managed by AWS Bedrock, we need to autowire the interface with the same name: ChatModel. We already configured all the chat options in the application.yaml Spring Boot will automatically create and configure the Instance (Bean) of ChatModel. To start a conversation with a particular prompt, we just need to write one line of code: ChatResponse call = chatModel.call(prompt); Let’s see what the whole service looks like: Java @Service public class ChatService { @Autowired private ChatModel chatModel; public String getResponse(String promptText) { Prompt prompt = new Prompt(promptText); ChatResponse call = chatModel.call(prompt); return call.getResult().getOutput().getContent(); } } That's it! We just need three lines of code to set up integration with the AWS Bedrock chat model. Isn't that amazing? Let’s dive deep into this code: As I previously mentioned, ChatModel bean is already configured by Spring Boot as we provided all the necessary info in the application.yaml file. We just need to ask Spring to inject it using Autowire annotation.We are creating a new instance of the Prompt object that provides the user with a prompt and executes the previously injected chat model. After receiving a response, we just return content, which is nothing but an actual chat response. Step 4. Creating Web Endpoint To easily reach our service, let’s create a simple GET endpoint. Java @RestController public class ChatController { @Autowired ChatService chatService; @GetMapping("/chat") public String getResponse(@RequestParam String prompt) { return chatService.getResponse(prompt); } } We are creating a new class annotated with @RestController.To reach our service, we need to inject it using field injection. To achieve this, we add our service and annotate it with @Autowire.To create an endpoint, we introduce a new method and annotate it with @GetMapping("/chat"). This method does nothing but receive a user prompt and pass it to the service we created in the previous step. Step 5. Run Our Application To start our application, we need to run the following command: mvn spring-boot:run When the application is running, we may check the result by executing the following curl with any prompt you want. I used the following: What is the difference between Spring, String, and Swing? Don't forget to add %20 instead of whitespaces if you are using the terminal for calling your endpoint: Shell curl --location 'localhost:8080/chat?prompt=What%20is%20the%20difference%20between%20Spring%2C%20String%20and%20Swing%3F' After executing, wait a few seconds as it takes some time for Amazon Bedrock to generate a response, and voila: Plain Text Spring is a framework for building enterprise-ready Java and Spring Boot applications. It offers a comprehensive set of tools and libraries to streamline development, reduce boilerplate code, and promote code reusability. Spring is known for its inversion of control (IoC) container, dependency injection, and aspect-oriented programming (AOP) features, which help manage the complexity of large applications. It also provides support for testing, caching, and asynchronous programming. Spring applications are typically built around the model-view-controller (MVC) architectural pattern, which separates the presentation, business logic, and data access layers. Spring applications can be deployed on a variety of platforms, including servlet containers, application servers, and cloud environments. String is a data type that represents a sequence of characters. It is a fundamental building block of many programming languages, including Java, and is used to store and manipulate text data. Strings can be accessed using various methods, such as length, index, concatenation, and comparison. Java provides a rich set of string manipulation methods and classes, including StringBuffer, StringBuilder, and StringUtils, which provide efficient ways to perform string operations. Swing is a graphical user interface (GUI) library for Java applications. It provides a set of components and APIs that enable developers to create and manage user interfaces with a graphical interface. Swing includes classes such as JFrame, JPanel, JButton, JTextField, and JLabel Step 6: Give More Flexibility in Model Configuration (Optional) In the second step, we configured the default behavior to our model and provided all the necessary configurations in the application.yaml file. But can we give more flexibility to our users and let them provide their configurations? The answer is yes! To do this, we need to use the ChatOptions interface. Here is an example: Java public String getResponseFromCustomOptions(String promptText) { ChatOptions chatOptions = new DefaultChatOptionsBuilder() .model("amazon.titan-text-express-v1") .topK(10) .temperature(0.1) .build(); return ChatClient.create(this.chatModel) .prompt(promptText) .options(chatOptions) .call() .content(); } To achieve this, we need to build options programmatically using all the configs we set up in application.yaml and provide these options into another interface called ChatClient. You may find more configuration options in Spring AI docs. Model: The model ID to use. You can use the supported models and model features Temperature: Controls the randomness of the output. Values can range over [0.0,1.0]TopK: Number of token choices for generating the next token. Conclusion Spring AI is a great tool that helps developers smoothly integrate with different AI models. As of writing this article, Spring AI supports a huge variety of chat models, including but not limited to Open AI, Ollama, and Deepseek AI. I hope you found this article helpful and that it will inspire you to explore Spring AI more deeply.

By Danil Temnikov CORE

Recommender systems serve as the backbone of e-commerce, streaming platforms, and online marketplaces, enabling personalized user experiences by predicting preferences and suggesting items based on historical interactions. They are built using explicit and/or implicit feedback from users. Explicit feedback includes direct user inputs, such as ratings and reviews, which provide clear indications of preference but are often sparse. Implicit feedback, such as clicks, views, purchase history, and dwell time, is more abundant but requires specialized algorithms to interpret user intent accurately. In contrast to conventional supervised learning tasks, recommender systems often grapple with implicit feedback, extreme data sparsity, and high-dimensional user-item interactions. These characteristics distinguish them from traditional regression or classification problems. The Netflix Prize competition was a milestone in this field, showcasing the superiority of latent factor models with matrix factorization over heuristic-based or naïve regression approaches. This article examines why standard regression models fall short in recommendation settings and outlines best practices for designing effective collaborative filtering systems. Problem Definition The core of the recommendation problem lies in the user-item matrix, denoted as Y, where Yui represents the rating assigned by user u to item i. In real-world datasets, this matrix is typically sparse, i.e., with a majority of entries missing. For instance, in the Netflix Prize dataset, each movie was rated by approximately 5,000 out of 500,000 users, resulting in a predominantly empty matrix (MIT 2025). This sparsity poses a significant challenge. Furthermore, the prevalence of implicit feedback (e.g., clicks, views) over explicit ratings adds another layer of complexity to generating accurate recommendations. Why Traditional Regression Struggles in Recommender Systems? For instance, in a movie recommendation system, a naive approach would be to treat the task as a regression problem, using features such as movie and user metadata, e.g., genre, actors, director, release year, and user preferences, to predict unknown user ratings. However, this approach has several limitations: Feature selection. Supervised learning depends on well-defined input features. However, in such problems, the determining factors — such as user preferences — are often hidden, difficult to engineer, and challenging to quantify.Sparse data and missing interactions. The user-item matrix in recommendation systems is inherently sparse, with most entries missing. This sparsity makes direct regression on raw ratings impractical.The cold start problem. New users and items often lack sufficient historical data for accurate predictions. For example, a new movie may not have enough ratings to assess its popularity, and new users may not have rated enough items to discern their preferences. Imputing missing ratings is also not a viable solution, as it fails to capture the behavioral context necessary for accurate recommendations. This presents a need for an alternative approach that does not rely solely on predefined item features. Collaborative filtering addresses these limitations by leveraging user-item interactions to learn latent representations, making it one of the most effective techniques in modern recommender systems. Collaborative Filtering Collaborative filtering operates on the principle that users who exhibit similar users are likely to share similar preferences. Unlike supervised regression techniques that rely on manually engineered features, collaborative filtering directly learns patterns from user-item interactions, making it a powerful and scalable approach for personalized recommendations. K-Nearest Neighbors (KNN) KNN, a supervised learning classifier, can be utilized for collaborative filtering. It provides recommendations for a user by looking at feedback from similar users. In this method, given a similarity function, S(u,v), between two users, u and v, a user’s rating for an item can be estimated as a weighted average of the ratings of their nearest neighbors. Common similarity measures include: Cosine similarity. Measures the cosine of the angle between the preference vectors of two users. It is particularly useful when user ratings are sparse and lack an inherent scale.Pearson correlation. Adjusts for differences in individual rating biases, making it more reliable when users have different rating scales. However, the effectiveness of KNN is limited by its dependence on the choice of similarity measure. Matrix Factorization Matrix factorization is a powerful technique for recommendation systems that decomposes the sparse user-item matrix Y into two lower-dimensional matrices, U and V, such that: Y≈UV U represents user-specific latent factors, and V represents item-specific latent factors. These latent factors capture the underlying features determining user preferences and item characteristics, enabling more accurate predictions even in the presence of missing data. Matrix factorization can be implemented with techniques such as singular value decomposition and alternating least squares. Best Practices for Collaborative Filtering Data Preprocessing Data preprocessing steps include handling missing values, removing duplicates, and normalizing data. Scalability As the size of the user-item matrix grows, computational efficiency becomes a concern. Approximate nearest neighbors or alternating least squares are preferred for handling large datasets. Diversity in Recommendation A good recommender system should also prioritize diversity, i.e., recommend a variety of items, including novel or unexpected choices, which can enhance user satisfaction and engagement. Handling Implicit Feedback In many real-world scenarios, explicit user ratings are scarce, and systems must rely on implicit feedback (e.g., clicks, views, or purchase history). Specialized algorithms like Weighted Alternating Least Squares are designed to handle implicit feedback effectively. These methods interpret user behavior as indicators of preference, enabling accurate predictions even without explicit ratings. Addressing the Cold Start Problem Recommendations for new users or items with limited or no interaction data is a challenge that can addressed by: Hybrid Models Combining collaborative filtering with content-based filtering or metadata-based approaches can effectively address the cold start problem. For example, if a new item lacks sufficient ratings, the system can use its metadata, e.g., genre, actors, or product descriptions, to recommend it based on similarity to other items. Similarly, for new users, demographic information or initial preferences can be used to bootstrap recommendations. Transfer Learning Transfer learning is a powerful technique for leveraging knowledge from related domains or user groups to improve recommendations for new users or items. For instance, in industries like healthcare or e-commerce, where user-item interactions may be sparse, transfer learning can apply insights from a data-rich domain to enhance predictions in a data-scarce one. Active Learning Active learning techniques can help gather targeted feedback from new users or for new items. By strategically prompting users to rate or interact with specific items, the system can quickly build a profile and improve recommendations. This approach is suited for scenarios where user engagement is high but initial data is sparse. Default Recommendations For new users or items, default recommendations based on popular or trending items can serve as a temporary solution until sufficient data is collected. While not personalized, this approach ensures that users receive relevant content while the system learns their preferences over time. Collaborative filtering is a powerful tool for building recommendation systems. By following best practices of proper data preprocessing, regularization, and evaluation and leveraging advanced techniques like hybrid models and transfer learning, practitioners can create robust and scalable recommender systems that deliver accurate, diverse, and engaging recommendations. References Massachusetts Institute of Technology (MIT). (n.d.) MITx 6.86x: Machine Learning with Python - From Linear Models to Deep Learning [Accessed 23 Feb. 2025].

By Salman Khan CORE