Java-Java Script123uo00es

Title Page Java 9 Programming By Example Your guide to software development Peter Verhas BIRMINGHAM - MUMBAI Java 9 Programming By Example Copyright © 2017 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: April 2017 Production reference: 1240417 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78646-828-4 www.packtpub.com Credits Author Copy Editors Peter Verhas Muktikant Garimella Zainab Bootwala Reviewer Project Coordinator Jeff Friesen Ulhas Kambali Commissioning Editor Proofreader Kunal Parikh Safis Editing Acquisition Editor Indexer Denim Pinto Mariammal Chettiyar Content Development Editor Graphics Nikhil Borkar Abhinash Sahu Technical Editor Production Coordinator Hussain Kanchwala Melwyn Dsa About the Author Peter Verhas is a senior software engineer and software architect having electrical engineering and economics background from TU Budapest (MsC) and PTE Hungary (MBA), and also studied at TU Delft and TU Vienna. He created his first programs in 1979, and since then he has authored several open source programs. He has worked in several positions in the telecommunications and finance industries and was the CIO of the Hungarian start-up index.hu during its early days. Peter works for EPAM Systems in Switzerland, participating in software development projects at various customer sites, and he supports talent acquisition by interviewing candidates, training programs for developers, and internal mentoring programs. You can follow Peter on Twitter at @verhas, LinkedIn, and GitHub, or read his technical blog, Java Deep, at http://javax0.wordpress.com. Acknowledgement is the section of a book that everybody ignores by turning the pages. This time, this section is a bit different. I will mention a few people and their roles in the making of this book but, at the same time, I will explain why and how it is important to rely on people, being a software developer. Doing professional work is not possible without having a life. It is quite obvious if you take that literally, but it is just as true figuratively. If you do not find the balance between your personal and professional life, you will burn out and will not operate professionally. This is the place to mention my family, my parents whom I am lucky to still have around, my wife, and my already adult kids who never stopped believing in me being able to do this work, who know more than well what a hypocrite I am, advocating personal-professional life balance, and who continually pushed me closer to this equilibrium point in life so that I could keep performing professionally. For professional work, coworkers are almost as important as family support. It is important that you support your colleagues as much as you ask them for their support. You learn a lot from books and from experience, but you learn the most from other people. Pay attention to senior developers. You can, however, learn just as much from juniors. No matter how ace you are, from time to time, a rookie may shed light on a topic. During the years, I learned a lot from juniors who brought a fresh view to the table, asking shocking questions that were absolutely valid. I cannot name each and every junior who aided my work with fresh out-of-the-box thinking. I can, and should, however, name some peer professionals who actively participated in the creation of this book with their advice, discussions, and suggestions. I should certainly mention Károly Oláh who was very enthusiastic about my project, and he represented, supported, and encouraged the idea inside EPAM systems. He actively discussed with the upper management that the support for writing a book well fits the innovation line and development of the company, and the people who work together. Without the official support from the company providing extra time for the task, I would not have been able to create this book. Good company attracts good people who are clever and also good to work with. I had many discussions about the book, topics, and how to explain certain aspects with my fellow EPAMers: Krisztián Sallai, Peter Fodor, Sándor Szilágyi, Mantas Aleknavicius, Gábor Lénard, and many others. I will separately mention István Attila Kovács from our Budapest office with whom I discussed Chapter 5 in detail, and who gave me very valuable feedback about the topic. If he does not know something about Java parallel computing, then that something does not exist. As a summary and takeaway for the patient reader who read this section till the end, technology, knowledge, skills, and experience are extremely important for being a professional Java 9 developer, but it is the people who really matter. About the Reviewer Jeff Friesen is a freelance author and software developer who has taught Java in the classroom and by writing numerous articles and books since the late 1990s. He holds a Bachelor of Science degree in Computer Science and Mathematics. Prior to freelancing, Jeff worked for telecommunications, investment, and software development companies. Jeff freelances as a Java author and software developer. Jeff has written Java I/O, NIO and NIO.2 and Java Threads and the Concurrency Utilities for Apress. Full details are available on his website (http://javajeff.ca/cgi-bin/makepage.cgi?/books). I thank Nitin Dasan for the opportunity to tech review this book. I also thank Ulhas Kambali for assisting me with the tech review process. www.PacktPub.com For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. https://www.packtpub.com/mapt Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career. Why subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via a web browser Customer Feedback Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/178646828X/. If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products! Table of Contents Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Errata Piracy Questions 1. Getting Started with Java 9 Getting started with Java Installing Java Installation on Windows Installation on MAC OS X Installation on Linux Setting JAVA_HOME Executing jshell Looking at the byte code Packaging classes into a JAR file Managing the running Java application Using an IDE NetBeans Eclipse IntelliJ IDE services IDE screen structure Editing files Managing projects Build the code and run it Debugging Java Summary 2. The First Real Java Program - Sorting Names Getting started with sorting Bubble sort Getting started with project structure and build tools Make Ant Installing Ant Using Ant Maven Installing Maven Using Maven Gradle Installing Gradle Setting up the project with Maven Coding the sort Understanding the algorithm and language constructs Blocks Variables Types Arrays Expressions Loops Conditional execution Final variables Classes Inner, nested, local, and anonymous classes Packages Methods Interfaces Argument passing Fields Modifiers Object initializers and constructors Compiling and running the program Summary 3. Optimizing the Sort - Making Code Professional The general sorting program A brief overview of various sorting algorithms Quick sort Project structure and build tools Maven dependency management Code the sort Creating the interfaces Creating BubbleSort Amending the interfaces Architectural considerations Creating unit tests Adding JUnit as dependency Writing the BubbleSortTest class Good unit tests A good unit test is readable Unit tests are fast Unit tests are deterministic Assertions should be as simple as possible Unit tests are isolated Unit tests cover the code Refactor the test Collections with wrong elements Handling exceptions Generics Test Driven Development Implementing QuickSort The partitioning class Recursive sorting Non-recursive sorting Implementing the API class Creating modules Why modules are needed What is a Java module Summary 4. Mastermind - Creating a Game The Game The model of the game Java collections Interface collection Set Hash functions Method equals Method hashCode Implementing equals and hashCode HashSet EnumSet LinkedHashSet SortedSet NavigableSet TreeSet List LinkedList ArrayList Queue Deque Map HashMap IdentityHashMap Dependency injection Implementing the game ColorManager The class color JavaDoc and code comments Row Table Guesser UniqueGuesser GeneralGuesser The Game class Creating an integration test Summary 5. Extending the Game - Run Parallel, Run Faster How to make Mastermind parallel Refactoring Processes Threads Fibers java.lang.Thread Pitfalls Deadlocks Race conditions Overused locks Starving ExecutorService ForkJoinPool Variable access The CPU heartbeat Volatile variables Synchronized block Wait and notify Lock Condition ReentrantLock ReentrantReadWriteLock Atomic classes BlockingQueue LinkedBlockingQueue LinkedBlockingDeque ArrayBlockingQueue LinkedTransferQueue IntervalGuesser ParallelGamePlayer Microbenchmarking Summary 6. Making Our Game Professional - Do it as a Webapp Web and network IP TCP/IP DNS The HTTP protocol HTTP methods Status codes HTTP/2.0 Cookies Client server and web architecture Writing servlets Hello world servlet Java Server Pages HTML, CSS, and JavaScript Mastermind servlet Storing state HTTP session Storing state on the client Dependency injection with Guice The MastermindHandler class Storing state on the server The GameSessionSaver class Running the Jetty web servlet Logging Configurability Performance Log frameworks Java 9 logging Logging practice Other technologies Summary 7. Building a Commercial Web Application Using REST The MyBusiness web shop Sample business architecture Microservices Service interface design JSON REST Model View Controller Spring framework Architecture of Spring Spring core Service classes Compiling and running the application Testing the application Integration test Application test Servlet filters Audit logging and AOP Dynamic proxy-based AOP Summary 8. Extending Our E-Commerce Application The MyBusiness ordering Setting up the project Order controller and DTOs Consistency checker Annotations Annotation retention Annotation target Annotation parameters Repeatable annotations Annotation inheritance @Documented annotations JDK annotations Using reflection Getting annotations Invoking methods Setting fields Functional programming in Java Lambda Streams Functional interfaces Method references Scripting in Java 9 Summary 9. Building an Accounting Application Using Reactive Programming Reactive... what? Reactive programming in a nutshell Reactive systems Responsive Resilient Elastic Message-driven Back-pressure Reactive streams Reactive programming in Java Implementing inventory Summary 10. Finalizing Java Knowledge to a Professional Level Java deep technologies Java agent Polyglot programming Polyglot configuration Polyglot scripting Business DSL Problems with polyglot Annotation processing Programming in the enterprise Static code analysis Source code version control Software versioning Code review Knowledge base Issue tracking Testing Types of tests Test automation Black box versus white box Selecting libraries Fit for the purpose License Documentation Project alive Maturity Number of users The "I like it" factor Continuous integration and deployment Release management Code repository Walking up the ladder Summary Preface Java drastically changed with the introduction of Java 8, and this change has been elevated to a whole new level with the new version, Java 9. Java has a well-established past, being more than 20 years old, but at the same time, it is new, functional, reactive, and sexy. This is a language that developers love, and at the same time, it is the number one choice of developer language for many enterprise projects. It is probably more lucrative to learn Java now than ever before, starting with Java 9. We encourage you to start your professional developer career by learning Java 9, and we have done our best in this book to help you along this road. We assembled the topics of the book so that it is easy to start, and you can feel the things working and moving very soon. At the same time, we have tried to reach very far, signaling the road ahead for a professional developer. The sands of time kept moving, and I discovered functional programming. I could very well see why writing side-effect-free code worked! I was hooked and started playing with Scala, Clojure, and Erlang. Immutability was the norm here. However, I wondered how traditional algorithms would look in a functional setting and started learning about it. A data structure is never mutated in place. Instead, a new version of the data structure is created. The strategy of copy and write with maximized sharing was an intriguing one! All that careful synchronization is simply not needed! The languages come equipped with garbage collection. So, if a version is not needed anymore, runtime would take care of reclaiming the memory. All in good time, though! Reading this book will help you see that we need not sacrifice algorithmic performance while avoiding in-place mutation! What this book covers Getting Started with Java 9, gives you a jump start in Java, helping you install it on your computer and run your first interactive programs using the new Jshell. Chapter 1, The First Real Java Program - Sorting Names, teaches you how to create a development project. This time, we will create program files and compile the code. Chapter 2, Optimizing the Sort - Making Code Professional, develops the code further so that the code is reusable and not only a toy. Chapter 3, Mastermind - Creating a Game, is when some fun starts. We develop a game application that is interesting and not as trivial as it first seems, but we will do it. Chapter 4, Extending the Game - Run Parallel, Run Faster, shows you how to utilize the multi-processor capabilities of modern architecture. This is a very important chapter that details technologies, that only a few developers truly understand. Chapter 5, Making Our Game Professional - Do it as a Webapp, transforms the user interface from command-line to web browser-based, delivering better user experience. Chapter 6, Building a Commercial Web Application Using REST, takes you through the development of an application that has the characteristics of many commercial applications. We will use the standard REST protocol which has gained ground in enterprise computing. Chapter 7, Extending Our E-Commerce Application, helps you develop the application further utilizing modern language features such as scripting and lambda expressions. Chapter 8, Building an Accounting Application Using Reactive Programming, teaches you how to approach some problems using reactive programming. Chapter 9, Finalizing Java Knowledge to a Professional Level, gives a bird's-eye view of developer topics that play an important role in the life of a Java developer, and which will guide you further in working as a professional developer. Chapter 10, What you need for this book To immerse into the content of this book and to soak up most of the skills and knowledge, we assume that you already have some experience with programming. We do not assume too much but hope that you already know what a variable is, that computers have memory, disk, network interfaces, and what they generally are. In addition to these basic skills, there are some technical requirements to try out the code and the examples of the book. You need a computer—something that is available today and can run Windows, Linux, or OSX. You need an operating system and, probably, that is all you need to pay for. All other tools and services that you will need are available as open source and free of charge. Some of them are also available as commercial products with an extended feature set, but for the scope of this book, starting to learn Java 9 programming, those features are not needed. Java, a development environment, build tools, and all other software components we use are open source. Who this book is for This book is for anyone who wants to learn the Java programming language. You are expected to have some prior programming experience with another language, such as JavaScript or Python, but no knowledge of earlier versions of Java is assumed. Conventions In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "The following function f has a side effect, though." A block of code is set as follows: package packt.java9.by.example.ch03; public interface Sort { void sort(SortableCollection collection); } If there is a line (or lines) of code that needs to be highlighted, it is set as follows: id=123 title=Book Java 9 by Example description=a new book to learn Java 9 weight=300 width=20 height=2 depth=18 New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Clicking the Next button moves you to the next screen." Warnings or important notes appear in a box like this. Tips and tricks appear like this. Reader feedback Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors. Customer support Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase. Downloading the example code You can download the example code files for this book from your account at https://github.com/PacktPublishing/Java -9-Programming-By-Example. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps: 1. 2. 3. 4. 5. 6. 7. Log in or register to our website using your e-mail address and password. Hover the mouse pointer on the SUPPORT tab at the top. Click on Code Downloads & Errata. Enter the name of the book in the Search box. Select the book for which you're looking to download the code files. Choose from the drop-down menu where you purchased this book from. Click on Code Download. Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: WinRAR / 7-Zip for Windows Zipeg / iZip / UnRarX for Mac 7-Zip / PeaZip for Linux The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Java-9Programming-By_Example. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out! Errata Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/ submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section. Piracy Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content. Questions If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem. Getting Started with Java 9 You want to learn Java and you have a good reason for it. Java is a modern and well-established application programming language, which is widely used in many industries, be it telecommunication, finance, or something else. Java developer positions are the most numerous and, probably, the best paid. This, among other things, makes the language lucrative for young professionals to learn. On the other hand, this is not without reason. Java language, the tools, and the whole infrastructure around it is complex and compound. Becoming a Java professional does not happen in a day or week; it is a work of many years. To be a Java expert, you need to know not only about the programming language but also about objectoriented programming principles, open source libraries, application servers, network, databases, and many other things that you can become an expert in. Nevertheless, learning the language is an absolute must that all other practices should build on. Through this book, you will be able to learn Java version 9 and a bit more. In this chapter, you will be introduced to the Java environment and given step-by-step instructions on how to install it, edit sample code, compile, and run Java. You will get acquainted with the basic tools that help development, be they are a part of Java or are provided by other vendors. We will cover the following topics in this chapter: Introduction to Java Installing Windows, Linux, and Mac OS X Executing jshell Using other Java tools Using integrated development environment Getting started with Java It is like going through a path in a forest. You can focus on the gravel of the road but it is pointless. Instead, you can enjoy the view, the trees, the birds, and the environment around you, which is more enjoyable. This book is similar as I won't be focusing only on the language. From time to time, I will cover topics that are close to the road and will give you some overview and directions on where you can go further after you finish this book. I will not only teach you the language but also talk a bit about algorithms, object-oriented programming principles, tools that surround Java development, and how professionals work. This will be mixed with the coding examples that we will follow. Lastly, the final chapter will be fully devoted to the topic, what to learn next and how to go further to become a professional Java developer. By the time this book gets into print, Java will have completed 22 years. http://www.oracle.com/technetwork/java/jav ase/overview/javahistory-index-198355.html. The language has changed a lot during this period and got better. The real question to ask is not how long has it been here, but rather how long will it stay? Is it still worth learning this language? There are numerous new languages that have been developed since Java was born (http://blog.takipi.com/java-vs-net-vs-python-vs-ruby-vs-node-js-who-reigns-the-job-market/). These languages are more modern and have functional programming features, which, by the way, Java has also had since version 8. Many say that Java is the past—the future is Scala, Swift, Go, Kotlin, JavaScript, and so on. You can add many other languages to this list, and for each, you can find a blog article that celebrates the burial of Java. There are two answers to this concern-one is a pragmatic business approach, the other is more engineering: Considering that COBOL is still actively used in the finance industry and COBOL developers are perhaps better paid than Java developers, it is not too risky to say that as a Java developer, you will find positions in the next 40 years. Personally, I would bet more than a 100 years, but considering my age, it will not be fair predicting more than 20 to 40 years ahead. Java is not only a language; it is also a technology that you will learn a bit about from this book. The technology includes the Java Virtual Machine (JVM), which is usually referred to as JVM, and gives the runtime environment for many languages. Kotlin and Scala, for example, cannot run without JVM. Even if Java will be adumbrated, JVM will still be a number one player in the enterprise scene. To understand and learn the basic operation of JVM is almost as important as the language itself. Java is a compiled and interpreted language. It is a special beast that forges the best of both worlds. Before Java, there were interpreted and compiled languages. Interpreted languages are read from the source code by the interpreter and then the interpreter executes the code. In each of these languages, there is some preliminary lexical and syntax analysis step; however, after that, the interpreter, which, as a program itself, is executed by the processor and the interpreter continuously, interprets the program code to know what to do. Compiled languages are different. In such a case, the source code is compiled to binary (.exe file on Windows platforms), which the operating system loads and the processor directly executes. Compiled programs usually run faster, but there is usually a slower compilation phase that may make the development slower, and the execution environment is not so flexible. Java combined the two approaches. To execute a Java program, the Java source code has to be compiled to the JVM byte code (.class file), which is loaded by JVM and is interpreted or compiled. Hmm... is it interpreted or compiled? The thing that came with Java is the Just in Time (JIT) compiler. This makes the phase of the compilation that is calculation-intensive and the compilation for compiled languages relatively slow. JVM first starts to interpret the Java byte code and, while doing that, it keeps track of execution statistics. When it gathers enough statistics about code executions, it compiles to native code (for example, x86 code on an Intel/AMD platform) for direct execution of the parts of code that are executed frequently and keeps interpreting the code fragments that are rarely used. After all, why waste expensive CPU time to compile some code that is hardly ever used? (For example, code that reads configuration during startup and does not execute again unless the application server is restarted.) Compilation to the byte code is fast and code generation is done only for the segments that pay off. It is also interesting that JIT uses the statistics of the code execution to optimize the code. If, for example, it sees that some conditional branch is executed in 99% of the cases and the other branch is executed only in 1%, then it will generate native code that runs fast, thus favoring the frequent branch. If the behavior of that part of the program changes by time and the statistic shows that the ratios changed, the JIT automatically recompiles the byte code from time to time. This is all automatic and behind the scenes. In addition to the automatic compilation, there is also an extremely important feature of JVM-it manages the memory for the Java program. The execution environment of modern languages do that and Java was the first mainstream language that had an automatic garbage collection (GC). Before Java, I was programming in C for 20 years and it was a great pain to keep track of all memory allocation and not to forget to release the memory when the program no longer needed it. Forgetting memory allocation at a single point in the code and the long running program was eating up all memory slowly. Such problems practically ceased to exist in Java. There is a price that we have to pay for it—GC needs processor capacity and some extra memory, but that is something we are not short of in most of the enterprise applications. Some special programs, like real-time embedded systems that control the brakes of a heavyduty lorry may not have that luxury. Those are still programmed in assembly or C. For the rest of us, we have Java, and though it may seem strange for many professionals, even almost-real-time programs, such as high-frequency trading applications, are written in Java. These applications connect through the network to the stock exchange and they sell and buy stocks responding to market change in milliseconds. Java is capable of doing that. The runtime environment of Java that you will need to execute a compiled Java code, which also includes the JVM itself, contains code that lets Java programs access the network, files on disks, and other resources. To do this, the runtime contains high-level classes that the code can instantiate, execute, and which do the low-level jobs. You will also do this. It means that the actual Java code does not need to handle IP packets, TCP connections, or even HTTP handling when it wants to use or provide a REST service in some microservices architecture. It is already implemented in the runtime libraries, and all the application programmer has to do is to include the classes in the code and use the APIs they provide on an abstraction level that matches the program. When you program in Java, you can focus on the actual problem you want to solve, which is the business code and not the low-level system code. If it is not in the standard library, you will find it in some product in some external library, and it is also very probable that you will find an open source solution for the problem. This is also a strong point of Java. There is a vast number of open source libraries available for all the different purposes. If you cannot find a library fitting your problem if you start to code some low-level code, then probably you are doing something wrong. There are topics in this book that are important, such as class loaders or reflection, not because you have to use them every day but rather because they are used by frameworks, and knowing them helps understand how these frameworks work. If you cannot solve your problem without using reflection or writing your own class loader or program multithread directly, then you probably chose the wrong framework. There is almost certainly a good one: Apache project, Google, and many other important players in the software industry publish their Java libraries as open source. This is also true for multithread programming. Java is a multithread programming environment from the very beginning. The JVM and the runtime supports programs that execute the code. The execution runs parallel on multiple threads. There are runtime language constructs that support parallel executing programs starting at the very low level to high abstraction. Multithread code utilizes the multicore processors, which are more effective. These processors are more and more common. 20 years ago, only high-end servers had multiple processors and only Digital Alpha processors had 64-bit architecture and CPU clock above 100 MHz. 10 years ago, multiprocessor structure was common on the server side, and about 5 years ago, multicore processors were on some desktops and on notebooks. Today, even mobile phones have them. When Java was started in 1995, the geniuses who created it had seen this future. They envisioned Java to be a write once, run anywhere language. At that time, the first target for the language was applet running in the browser. Today, many think (and I also share this opinion) that applets were a wrong target, or at least things were not done in the right way. As for now, you will meet applets on the Internet less frequently than Flash applications or dinosaurs. However, at the same time, the Java interpreter was also executing server and client applications without any browser; furthermore, as the language and the executing environment developed, these application areas became more and more relevant. Today, the main use of Java is enterprise computing and mobile applications mainly for the Android platform; for the future, the use of the environment is growing in embedded systems as the Internet of things (IoT) comes more and more into picture. Installing Java To develop, compile, and execute Java programs, you will need the Java execution environment. As the operating systems that we usually use for software development do not contain the language preinstalled, you will have to download it. Although, there is multiple implementation of the language, I recommend that you download the official version of the software from Oracle. The official site for java is http://java.co m and this is the site from where the latest release of the language can be downloaded. At the time of writing this book, the 9th version of Java is not yet released. An early pre-release version is accessible via http://jdk9.java.net/download. Later the release versions will also be available from here. What you can download from here is a so called early access version of the code that is available only to experiment with it, and no professionals should use it for real professional purposes On the page, you have to click on the radio button to accept the license. After that, you can click on the link that directly starts the download of the installation kit. The license is a special early access license version that you, as a professional, should carefully read, understand, and accept only if you are agreeable with the terms. There is a separate installation kit for Windows 32 and 64 bit systems, Mac OS X, Linux 32 and 64 bit versions, Linux for ARM processor, Solaris for SPARC processor systems, and Solaris x86 versions. As it is not likely that you will use Solaris, I will detail the installation procedure only for Windows, Linux, and Mac OS X. In later chapters, the samples will always be Mac OS X, but since Java is a write once, run anywhere language, there is no difference after the installation. The directory separator may be slanted differently, the classpath separator character is a semicolon on Windows instead of a colon, and the look and feel of the Terminal or command application is also different. However, where it is important, I will try not to forget to mention it. To confuse you, the Java download for each of these operating system versions lists a link for the JRE and one for the JDK. JRE stands for Java Runtime Environment and it contains all the tools and executables that are needed to run Java programs. JDK is the Java Development Kit that contains all the tools and executables that are needed to develop Java programs including the execution of the Java program. In other words, JDK contains its own JRE. For now, all you need to do is download the JDK. There is one important point of the installation that is the same on each of the three operating systems that you have to be prepared for before the installation: to install Java, you should have administrative privileges. Installation on Windows The installation process on Windows starts by double clicking on the downloaded file. It will start the installer that will present you a welcome screen. Pressing the Next button we get a window where you can select the parts you want to install. Let's leave here the default selection, which means that we install all the downloaded parts of Java and press the button Next. The following window is where we can select the destination folder for the installation. As for now we do not change the directory selected by the installer. Press Next. Later, when you become a professional Java developer, you may decide to install Java to a different location but then you will already have to know what you are doing. You may need to click the Next button a few times and then the installer finishes. Provide the administrative password when asked and voila! Java is installed. This is really the very usual Windows installation process. The last step is to set the environment variable JAVA_HOME. To do that in Windows we have to open the control center and select the Edit environment variables for your account menu. This will open a new window that we should use to create a new environment variable for the current user. The name of the new variable has to be JAVA_HOME and the value should point to the installation directory of the JDK. This value on most of the systems is C:Program FilesJavajdk-9. This is used by many Java programs and tools to locate the Java runtime. Installation on MAC OS X In this section, we will take look at how to install Java step-by-step on an OS X platform. I will describe the installation process for the released version available at the time of writing this book. As for now, the Java 9 early access version is a bit tricky to install. It is probable that version Java 9 will have similar or the same install steps as Java 8 update 92 has. The OS X version of Java comes in the form of a .dmg file. This is a packaging format of OS X. To open it, simply double click on the file in the Download folder where the browser saves it and the operating system will mount the file as a read-only disk image. There is only one file on this disk: the installation image. Double click on the file name or icon in the Finder application and the installation process will start. The first screen opening is a welcome screen. Click Continue and you will see the Summary page that displays what will be installed. It is not a surprise that you will see a standard Java installation. This time, the button is called Install. Click on it and you will see the following: This is the time when you have to provide the login parameters for the administrative user—a username and password. When provided, installation starts and, in a few seconds, you will see a Summary page. Click Close and you are ready. You have Java installed on your Mac. Optionally, you can dismount the installation disk and, sometime later, you can also delete the .dmg file. You will not need that, and in case you do, you can download it any time from Oracle. The last thing is to check whether the installation was okay. Proof of the pudding is eating it. Start a Terminal window and type java -version at the prompt and Java will tell you the version installed. On the next screenshot you can see the output on my workstation and also the Mac OS commands that are handy to switch between the different versions of Java: On the screenshot, you can see that I have installed the Java JDK 1.8u92 version and, at the same time, I also have a Java 9 early release installation, which I will use to test the new features of Java for this book. Installation on Linux There are several ways to install Java on Linux, depending on its flavor. Here, I will describe an installation method that works more or less the same way on all flavors. The one I used is Debian. First step is the same as in any other operating system: download the installation kit. In the case of Linux, you should select a package that has a tar.gz ending. This is a compressed archive format. You should also carefully select the package that matches the processor in your machine and the 32/64 bit version of the operating system. After the package is downloaded, you have to switch to root mode, issuing the su command. This the first command you can see on the screenshot that shows the installation commands. The tar command uncompressed the archive into a subfolder. In Debian, this subfolder has to be moved to /opt/jdk and the mv command is used for this purpose. The two update-alternatives command is Debianspecific. These tell the operating system to use this newly installed Java in case there is already an older Java installed. The Debian I was using to test and demonstrate the installation process on a virtual machine came with a 7 year old version of Java. The final step of the installation is the same as any other operating system: checking that the installation was successful in issuing the java -version command. In the case of Linux, this is even more important because the installation process does not check that the downloaded version matches the operating system and the processor architecture. Setting JAVA_HOME The JAVA_HOME environment variable plays a special role for Java. Even though the JVM executable, java.exe or java, is on the PATH (thus you can execute it by typing the name java without specifying directory on the Command Prompt) (Terminal), it is recommended that you use the correct Java installation to set this environment variable. The value of the variable should point to the installed JDK. There are many Javarelated programs, for example, Tomcat or Maven, that use this variable to locate the installed and currently used Java version. In Mac OS X, setting this variable is unavoidable. In OS X, the program that starts to execute when you type java is a wrapper that first looks at JAVA_HOME to decide which Java version to start. If this variable is not set, then OS X will decide on its own, selecting from the available installed JDK versions. To see the available versions, you can issue the following command: ~$ /usr/libexec/java_home -V Matching Java Virtual Machines (10): 9, x86_64: "Java SE 9-ea" /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home 1.8.0_92, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home 1.7.0_60, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home You will then get the list of installed JDKs. Note that the command is lowercase, but the option is capital. If you do not provide any options and argument to the program, it will simply return the JDK it thinks is the newest and most appropriate for the purpose. As I copied the output of the command from my Terminal window, you can see that I have quite a few versions of Java installed on my machine. The last line of the program response is the home directory of JDK, which is the default. You can use this to set your JAVA_HOME variable using some bash programming: export JAVA_HOME=$(/usr/libexec/java_home) You can place this file in your .bashrc file, which is executed each time you start Terminal application and thus JAVA_HOME will always be set. If you want to use a different version, you can use -v, with the lower case option this time, to the same utility, as follows: export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) The argument is the version of Java you want to use. Note that this versioning becomes: export JAVA_HOME=$(/usr/libexec/java_home -v 9) If you want to use Java JDK Early Access version and not 1.9, there is no explanation for the same—fact of life. Note that there is another environment variable that is important for Java-CLASSPATH. We will talk about it later. Executing jshell Now that we have spent a lot of time installing Java, it is time to get the fingers burnt a bit. As we are using Java 9, there is a new tool that helps developers to play around with the language. This is a ReadEval-Print-Loop (REPL) tool that many language toolsets contain and there were also implementations from Java, but version 9 is the first that contains this feature off the shelf. REPL is a tool that has interactive prompt and language commands that can be directly entered without editing some standalone file. The entered commands are executed directly and then the loop starts again, waiting for the user to type in the next command. This is a very effective tool to try out some language constructs without the delay of editing, compiling, and loading. The steps are automatically and transparently done by the REPL tool. The REPL tool in Java 9 is called jshell. To start it, just type its name. If it is not on the PATH, then type the full path to jshell that comes installed with Java 9, as shown in the following example: $ jshell | Welcome to JShell -- Version 9-ea | For an introduction type: /help intro jshell> The jshell starts up in an interactive way and the prompt it displays is jshell> to help you recognize that jshell is running and what you type is read by the program and not the operating system shell. As this is the first time you will start jshell, it tells you to type /help intro. Let's do it. It will print out a short text about what jshell is, as shown in the following code: jshell> /help intro | | intro | | The jshell tool allows you to execute Java code, getting immediate results. | You can enter a Java definition (variable, method, class, etc), like: int x = 8 | or a Java expression, like: x + x | or a Java statement or import. | These little chunks of Java code are called 'snippets'. | | There are also jshell commands that allow you to understand and | control what you are doing, like: /list | | For a list of commands: /help Okay, so we can type Java snippets and /list, but that is only one example of the available commands. We can hope for more information by typing /help, as shown in the following code: jshell> /help | Type a Java language expression, statement, or declaration. | Or type one of the following commands: | /list [<name or id>|-all|-start] | /edit <name or id> | /drop <name or id> | /save [-all|-history|-start] <file> ... ----- list the source you have typed edit a source entry referenced by name or id delete a source entry referenced by name or id Save snippet source to a file. What you get is a long list of commands. Most of it is not presented here to save paper and your attention. We will use many of these commands on our journey through the next few pages. Let's start with a small Java snippet that is the ageless Hello World example: jshell> System.out.println("Hello World!") Hello World! This is the shortest ever Hello World program in Java. Till Java 9, if you wanted to do nothing more than print out Hello World!, you had to create a program file. It had to contain the source code of a class including the public static main method, which contained the one line we had to type in with Java 9 jshell. It was cumbersome just for a simple printout of sample code. Now it is much easier and jshell is also lenient, forgiving us the missing semicolon at the end of the line. The next thing we should try is declaring a variable, as follows: jshell> int a = 13 a ==> 13 jshell> We declared a variable, named a, and assigned the value to it-13. The type of the variable is int, which is an abbreviation for integer types in Java. Now we have this variable already in our snippet, so we can print it out if we want to as shown: jshell> System.out.println(a) 13 It is time to write something more complex into jshell than a one liner. jshell> void main(String[] args){ ...> System.out.println("Hello World") ...> } | Error: | ';' expected | System.out.println("Hello World") | The jshell recognizes that this is not a one-liner and that it cannot process what we typed so far, when we press Enter at the end of the first line, and it signals that it expects more characters from us, so it displays ...> as a continuation prompt. We type in the commands that make up the whole hello world main method, but this time jshell does not let us miss the semicolon. That is allowed only in the case of one-line snippets. As jshell is interactive, it is easy to correct the mistake; press the up arrow key a few times to get back the previous lines and, this time, add the semicolon at the end of the second line: jshell> void main(String[] args){ ...> System.out.println("Hello World"); ...> } | created method main(String[]) This method was created for us as a snippet and now we can call it: jshell> main(null) Hello World And it works. You can list all the snippets that were created, as follows: jshell> /list 1 : System.out.println("Hello World!") 2 : int a = 13; 3 : System.out.println(a) 4 : void main(String[] args){ System.out.println("Hello World"); } And, as we want to go on writing a full Java version of hello world, we can save our work from jshell to a file, as follows: jshell> /save HelloWorld.java Finally, we exited from jshell by typing /exit. As you get back to the system prompt, type cat (or type HelloWorld.java on Windows) to see the content of the file. It is as follows: HelloWorld.java $ cat HelloWorld.java System.out.println("Hello World!") int a = 13; System.out.println(a) void main(String[] args){ System.out.println("Hello World"); } The file contains all the snippets that we typed in one after the other. If you think that you have messed up the shell with lots of variables and code snippets that you do not need anymore, you can issue the /reset command: jshell> /reset | Resetting state. After this command, the jshell is as clean as when it was started earlier jshell> /list jshell> Listing just does not produce anything as we deleted it all. Fortunately, we saved the state of jshell to a file and we can also load the content of the file issuing the /open command: jshell> /open HelloWorld.java Hello World! 13 It loads the line from the file and executes it just as the characters were typed into the Command Prompt. You may recall that the /list command printed a number in front of each snippet. We can use it to edit the snippets individually. To do so, issue the /edit command followed by the number of the snippet: jshell> /edit 1 You may recall that the first command we entered was the System.out.println system call that prints out the argument to the console. When you press Enter after the /edit 1 command, you do not get the prompt back. Instead, jshell opens a separate graphical editor that contains the snippet to edit as shown in the following image: Edit the text in the box so that it will look like this: printf("Hello World!") Click on Accept and then Exit. When you click on Accept, the Terminal will execute the snippet and display the following result: Hello World! The method that we used, printf, stands for formatted printing. This may be well known from many other languages. It was first introduced by the C language and though cryptic, the name survived. This is also part of the standard Java class, PrintStream, just like println. In case of println, we had to write System.out in front of the method name. In case of printf, we did not. Why? The reason is that jshell defines a few snippets that are automatically loaded when jshell starts or resets. You can see these if you issue the /list command with the -start option, as follows: jshell> /list -start s1 s2 s3 s4 s5 s6 s7 s8 : : : : : : : : import java.util.*; import java.io.*; import java.math.*; import java.net.*; import java.util.concurrent.*; import java.util.prefs.*; import java.util.regex.*; void printf(String format, Object... args) { System.out.printf(format, args); } These predefined snippets help the use of jshell. Most of the users will import these classes, and to ease the print to screen, it defines a method snippet that happens to have the name, printf, which is also the name of a method in the PrintStream class. If you want to list all the snippets you entered as well as the predefined snippets and also those that contained some error and thus were not executed, you can use the -all option to the /list command, as follows: jshell> /list -all ... s7 : import java.util.regex.*; ... 1 : System.out.println("Hello World!") ... e1 : System.out.println("Hello World!") int a = 14; 5 : System.out.println("Hello World!"); ... Some of the lines were deleted from the actual output for brevity. The lines that are preloaded are numbered with the s prefix. The snippets that contain an error have a number prefixed with e. If you want to execute some of the snippets again, you only have to type /n where n is the number of the snippet, as follows: jshell> /1 System.out.println("Hello World!") Hello World! You cannot re-execute the preloaded snippets or snippets that contained errors. There is no need for any of those anyway. Preloaded snippets declare some imports and define a snippet method; erroneous snippets do not execute because they are, well...erroneous. You need not rely on the number of jshell when you want to re-execute a snippet. When you already have a lot of snippets in your jshell session, listing them all would be too cumbersome; there is a shortcut to reexecute the last n-th snippet. You have to write /-n. Here, n is the number of the snippet counting from the last one. So, if you want to execute the very last snippet, then you have to write /-1. If you want to execute the one before the last one, you have to write /-2. Note that if you already typed /-1, then the last one is the re-execution of the last snippet and snippet number -2 will become number -3. Listing all the snippets can also be avoided in other ways. When you are interested only in certain types of snippets, you can have special commands. If we want to see only the variables that we defined in the snippets, then we can issue the /vars command, as follows: jshell> /vars | int a = 13 If we want to see only the classes, the command/types will do that: jshell> class s {} | created class s jshell> /types | class s Here, we just created an empty class and then we listed it. To list the methods that were defined in the snippets, the /methods command can be issued: jshell> /methods | printf (String,Object...)void | main (String[])void You can see in the output that there are only two methods, which are as follows: : This is defined in a preloaded snippet main: This, we defined printf If you want to see everything you typed, you have to issue the /history command for all the snippets and commands that you typed. (I will not copy the output here; I do not want to shame myself. You should try yourself and see your own history.) Recall that we can delete all the snippets issuing the /reset command. You can also delete snippets individually. To do so, you should issue the /drop n command, where n is the snipped number: jshell> /drop 1 | This command does not accept the snippet '1' : System.out.println("Hello World!") | See /types, /methods, /vars, or /list Oops! Something went wrong. There is nothing defined when snippet number 1 was executed and the /drop command actually drops the defined variable, type, or method. There is nothing to be dropped in the first snippet. But, if we reissue the /list command, we will get the following results: jshell> /list 1 2 3 4 : : : : System.out.println("Hello World!") int a = 13; System.out.println(a) void main(String[] args){ System.out.println("Hello World"); } We can see that we can drop the second or the fourth snippet, too: jshell> /drop 2 | dropped variable a jshell> /drop 4 | dropped method main(String[]) The jshell error message says to see the output of the /types, /methods, /vars, or /list commands. The problem with this is that /types, /methods, and /vars do not display the number of the snippet. This is most probably a small bug in the jshell prerelease version and may be fixed by the time the JDK is released. When we were editing the snippets, jshell opened a separate graphical editor. It may happen that you are running jshell using ssh on a remote server and where it is not possible to open a separate window. You can set the editor using the /set command. This command can set quite a few configuration options of the jshell. To set the editor to use the ubiquitous vi, issue the following command: jshell> /set editor "vi" | Editor set to: vi After this, jshell will open the snipped-in vi in the same Terminal window where you issue the /edit command. It is not only the editor that you can set. You can set the startup file, and the way jshell prints the feedback to the console after a command was executed. If you set the startup file, then the commands listed in the startup file will be executed instead of the builtin commands of jshell after the /reset command. This also means that you will not be able to use the classes directly that are imported by default and you will not have the printf method snippet, unless your own startup file contains the imports and the definition of the snippet. Create the sample.startup file with the following content: void println(String message) { System.out.println(message); } Starting up a new jshell and executing it is done as follows: jshell> /set start sample.startup jshell> /reset | Resetting state. jshell> println("wuff") wuff jshell> printf("This won't work...") | Error: | cannot find symbol | symbol: method printf(java.lang.String) | printf("This won't work...") | ^----^ The println method is defined but the printf method, which was defined in the default startup, is not. The feedback defines the prompt jshell prints and then waits for the input, the prompt for the continuation lines, and the message details after each command. There are predefined modes, which are as follows: Normal Silent Concise Verbose Normal is selected by default. If you issue /set feedback silent, then prompt becomes -> and jshell will not print details about the commands. The /set feedback concise code prints a bit more information and /set feedback verbose prints verbose information about the commands executed: jshell> /set feedback verbose | Feedback mode: verbose jshell> int z = 13 z ==> 13 | modified variable z : int | update overwrote variable z : int You can also define your own modes, giving a name to the new mode using the /set mode xyz command where xyz is the name of the new mode. After this, you can set prompt, truncation, and format for the mode. When the format is defined, you can use it the same way as the built-in modes. Last, but not least, the most important command of jshell is /exit. This will just terminate the program and you will return to the operating system shell prompt. Now, let's edit the HelloWorld.java file to create our first Java program. To do so, you can use vi, notepad, Emacs, or whatever is available on your machine and fits you. Later on, we will use some integrated development environment (IDE), NetBeans, Eclipse, or IntelliJ; however, for now, a simple text editor is enough. Edit the file so that the content will be as follows: public class HelloWorld { public static void main(String[] args){ System.out.println("Hello World"); } } To compile the source code to byte code, which is executable by JVM, we have to use the Java compiler named javac: javac HelloWorld.java This generates the java.class file in the current directory. This is a compiled code that can be executed as follows: $ java HelloWorld Hello World With this one, you have created and executed your first full Java program. You may still wonder what we were doing. How and why, I will explain it; but first, I wanted you to have a feeling that it works. The file we edited contained only the snippet and we deleted most of the lines, except the declaration of the main method and we inserted the declaration of the class around it. In Java, you cannot have standalone methods or functions, like in many other languages. Every method belongs to some class and every class should be declared in a separate file (well, almost, but for now, let's skip the exceptions). The name of the file has to be the same as the name of the class. The compiler requires this for public classes. Even for non-public classes we usually follow this convention. If you renamed the file from HelloWorld.java to Hello.java, the compiler will display an error when you try to compile the file with the new name. $ mv HelloWorld.java Hello.java ~/Dropbox/java_9-by_Example$ javac Hello.java Hello.java:2: error: class HelloWorld is public, should be declared in a file named HelloWorld.java public class HelloWorld { ^ 1 error So, let's move it back to the original name: mv . Hello.java HelloWorld.java The declaration of the class starts with the keyword class, then the name of the class, an opening curly brace, and lasts until the matching closing brace. Everything in between belongs to the class. For now, let's skip why I wrote public in front of the class and focus on the main method in it. The method does not return any value, therefore; the return value of it is void. The argument, named args, is a string array. When JVM starts the main method, it passes the command-line arguments to the program in this array. However, this time we do not use it. The main method contains the line that prints out Hello World. Now, let's examine this line a bit more. In other languages, printing something to the console requires only a print statement or a very similar command. I remember that some BASIC interpreters even allowed us to type ? instead of print because printing to the screen was so common. This has changed a lot during the last 40 years. We use graphical screens, Internet, and many other input and output channels. These days, it is not very common to write to the console. Usually, in professional large-scale enterprise applications, there is not even a single line that does that. Instead, we will direct the text to log files, send messages to message queues, and send requests and reply with responses over TCP/IP protocol. As this is so infrequently used, there is no reason to create a shortcut for the purpose in the language. After the first few programs, when you get acquainted with the debugger and logging possibilities, you will not print anything directly to the console yourself. Still, Java has features that let you send text directly to the standard output of a process the good old way, as it was invented originally for UNIX. This is implemented in a Java way where everything has to be an object or class. To get access to the system output, there is a class named System and it, among other things, has the following three variables: : This is the standard input stream out: This is the standard output stream err: This is the standard error stream in To refer to the output stream variable, because it is not in our class but in System, we will have to specify the class name so we will refer to it as System.out in our program. The type of this variable is PrintStream, which is also a class. Class and type are synonyms in Java. Every object that is of type PrintStream has a method named println that accepts a String. If the actual print stream is the standard output, and we are executing our Java code from the command line, then the string is sent to the console. The method is named main and this is a special name in Java programs. When we start a Java program from the command line, JVM invokes the method named main from the class that we specify on the command line. It can do that because we declared this method public so that anyone can see and invoke it. If it was private, it would be seen and callable only from within the same class, or classes, that are defined in the same source file. The method is also declared as static, which means that it can be invoked without an actual instance of the class that contains the methods. Using static methods is usually seen as not a good practice these days, unless they are implementing functions that cannot really ever be related to an instance, or have different implementations such as the functions in the java.lang.Math class; but, somewhere, the code execution has to start and the Java runtime will not usually create instances of classes for us automatically. To start the code, the command line should be as follows: java -cp . HelloWorld The -cp option stands for classpath. The classpath is a fairly complex idea for java but, for now, let's make it simple and say that it is a list of directories and JAR files that contain our classes. The list separator for the classpath is : (colon) on UNIX-like systems and ; (semicolon) on Windows. In our case, the classpath is the actual directory, as that is the place where the Java compiler created HelloWorld.class. If we do not specify classpath on the command line, Java will use the current directory as a default. That is the reason our program was working without the -cp option in the first place. Both java and javac handle many options. To get a list of the options type javac -help or java -help. We use the IDE to edit the code and, many times, to compile, build, and run it during development. The environment in this case sets the reasonable parameters. For production we use build tools that also support the configuration of the environment. Because of this, we rarely meet these command line options. Nevertheless, professionals have to understand their meanings at least and know where to learn their actual use in case it is needed. Looking at the byte code The class file is a binary file. The main role of this format is to be executed by the JVM and to provide symbolic information for the Java compiler when a code uses some of the classes from a library. When we compile our program that contains System.out.println, the compiler looks at the compiled .class files and not at the source code. It has to find the class named System, the field named out, and the method println. When we debug a piece of code or try to find out why a program does not find a class or method, we will need a way to look into the binary of the .class files. This is not an everyday task and it takes some advanced knowledge To do so, there is a decompiler that can display the content of a .class file in a more or less readable format. This command is called javap. To execute it, you can issue the following command: $ javap HelloWorld.class Compiled from "HelloWorld.java" public class HelloWorld { public HelloWorld(); public static void main(java.lang.String[]); } The output of the program shows that the class file contains Java class that has something called HelloWorld(); it seems to be a method having the same name as the class and it also contains the method we have written. The method that has the same name as the class is the constructor of the class. As every class in java can be instantiated, there is a need for a constructor. If we do not give one, then the Java compiler will create one for us. This is the default constructor. The default constructor does nothing special but returns a new instance of the class. If we provide a constructor on our own, then the Java compiler will not have bothered creating one. The javap decompiler does not show what is inside the methods or what Java code it contains unless we provide the -c option: $ javap -c HelloWorld.class Compiled from "HelloWorld.java" public class HelloWorld { public HelloWorld(); Code: 0: aload_0 1: invokespecial #1 // 4: return public static void main(java.lang.String[]); Code: 0: getstatic #2 // 3: ldc #3 // 5: invokevirtual #4 // 8: return } Method java/lang/Object."<init>":()V Field java/lang/System.out:Ljava/io/PrintStream; String hali Method java/io/PrintStream.println:(Ljava/lang/String;)V It is very cryptic and is not for ordinary humans. Only a few experts, who deal with the Java code generation, can fluently read that. But, to have a look at it helps you get a glimpse of what byte code means. It is something like a good old assembly. Although this is binary code, there is nothing secret in it: Java is open source, the class file format is well documented and debuggable for the experts. Packaging classes into a JAR file When you deliver a Java application, usually the code is packaged into JAR, WAR, EAR, or some other packaged format. We learn something again that seems to be obscure at first sight, but in reality, this is not that complex. They are all ZIP files. You can open any of these files using WinZip or some other zip manager that you have a license for. The extra requirement is that, for example, in the case of a JAR file, the archive should contain a directory named META-INF and inside it a file named MANIFEST.MF. This file is a text file and contains meta information in the format, which is as follows: Manifest-Version: 1.0 Created-By: 9-ea (Oracle Corporation) There can be a lot of other information in the file, but this is the minimum that the Java provided tool jar puts there if we package our class file into a jar issuing the following command: jar -cf hello.jar HelloWorld.class The -c option tells the JAR archiver to create a new JAR file and the option f is used to specify the name of the new archive. The one we specified here is hello.jar and the file added to it is the class file. The packaged JAR file can also be used to start the Java application. Java can read directly from JAR archives and load classes from there. The only requirement is that they are on the classpath. Note that you cannot put individual classes on the classpath, only directories. As JAR files are archives with an internal directory structure in them, they behave like a directory. Check that the JAR file was created using ls hello.jar and remove the rm HelloWorld.class class file just to ensure that when we issue the command line, the code is executed from the JAR file and not the class. $ java -cp hello.jar HelloWorld Hello World To see the content of the JAR file, however, it is recommended that you use the JAR tool and not WinZip even though that may be cozier. Real professionals use the Java tools to handle Java files. $ jar -tf hello.jar META-INF/ META-INF/MANIFEST.MF HelloWorld.class Managing the running Java application The Java toolset that comes with the JDK supports the execution and management of running Java applications as well. To have some program that we can manage while executing, we will need a code that runs not only for a few milliseconds but, while it runs, it also prints something to the console. Let's create a new program called HelloWorldLoop.java with the following content: public class HelloWorldLoop { public static void main(String[] args){ for( ;; ){ System.out.println("Hello World"); } } } The program contains a for loop. Loops allow repeated execution of a code block, and we will discuss them in Chapter 2, The First Real Java Program - Sorting Names. The loop we created here is a special one that never terminates but repeats the printing method call, printing Hello World until we kill the program by pressing Ctrl + c or issuing a kill command on Linux or on OSX, or terminate the program in the task manager under Windows. Compile and start it in one window and open another Terminal window to manage the application. The first command that we should get familiar with is jps. http://docs.oracle.com/javase/7/docs/technotes/tools/share/jps.ht ml It lists the Java processes that run on the machine, which are as follows: $ jps 21873 sun.tools.jps.Jps 21871 HelloWorldLoop You can see that there are two processes—one is the program we execute and the other is the jps program itself. Not surprisingly, the jps tool is also written in Java. You can also pass options to jps, which are documented on the web. There are many other tools and we will examine one of them, which is a very powerful and easy-to-use tool—Java VisualVM. VisualVM is a command-line graphical tool that connects to the running Java process and displays the different performance parameters. To start the VisualVM tool, you will issue the jvisualvm command without any parameters. Soon, a window will appear with an exploring tree on the left-hand side and a welcome pane on the right. The left side shows all the running Java processes under the branch named Local. If you double click on HelloWorldLoop, it will open the details of the process on the right pane. On the header tabs, you can select Overview, Monitor, Threads, Sampler, and Profiler. The first three tabs are the most important and give you a good view of what is happening in JVM regarding the number of threads, CPU usage, memory consumption, and so on. Using an IDE Integrated development environments are outstanding tools that help the development by offloading the mechanical tasks from the developer's shoulders. They recognize many of the programming errors as we type the code, help us find the needed library methods, display the documentation of the libraries, and provide extra tools for style checking, debugging, and so on. In this section, we will look at some IDEs and how to leverage the functions they provide. To get an IDE, you will have to download and install it. It does not come with the Java development tools because they are not part of the language environment. But, don't worry. They can be downloaded free of charge and are easy to install. They may be more complex to start up than a notepad editor, but even after a few hours of work, they will pay back the time you devote to learning them. After all, it is not without reason that no developer is coding Java in notepad or vi. The three topmost IDEs are NetBeans, Eclipse, and IntelliJ. All are available in community versions, which means that you need not pay for them. IntelliJ has a full version that you can also buy. The community edition will be usable for learning the language. In case you do not like IntelliJ, you can use Eclipse or NetBeans. These are all free of charge. Personally, I use the IntelliJ community edition for most of my projects and the screen samples that show an IDE in this book will feature this IDE. But, it does not necessarily mean that you have to stick to this IDE. In the developer community, there are topics that can be heavily debated. These topics are about opinions. Were they about facts the debate would easily be soon over. One such topic is: "Which is the best IDE?" It is a matter of taste. There is no definite answer. If you learn how to use one, you will like that and you will be reluctant to learn another one, unless you see that the other one is so much better. That is the reason developers love the IDE they use (or just hate, depending on their personality), but they keep using the same IDE usually for a long time. There is no best IDE. To download the IDE of your choice, you can visit either one of the following websites: for NetBeans http://www.eclipse.org/ for Eclipse https://www.jetbrains.com/idea/ for IntelliJ https://netbeans.org/ NetBeans NetBeans is supported by Oracle and is continuously developed. It contains components, such as the NetBeans profiler, that became part of the Oracle Java distribution. You may notice that when you start Visual VM and start the profiling, the Java process started has netbeans in its name. Generally, NetBeans is a framework to develop rich client applications and the IDE is only one application of the many that are built on top of the framework. It supports many languages, not only Java. You can develop PHP, C, or JavaScript code using NetBeans and have similar services for Java. For the support of different languages, you can download plugins or a special version of NetBeans. These special versions are available from the download page of the IDE and they are nothing more than the basic IDE with some preconfigured plugins. In the C package, the developers configure the plugins that are needed when you want to develop C; in the PHP version, they plugin for PHP. Eclipse Eclipse is supported by IBM. Similar to NetBeans, it is also a platform for rich client application and it is built around the OSGi container architecture, which itself is a topic that can fill a book like this. Most of the developers use Eclipse and, almost exclusively, it is the choice when developers create code for the IBM WebSphere application server. The Eclipse special version contains a developer version of WebSphere. Eclipse also has plugins to support different programming languages and also has different variations similar to NetBeans. The variations are plugins prepackaged with the basic IDE. IntelliJ The last one in the preceding enumeration is IntelliJ. This IDE is the only one that does not want to be a framework. IntelliJ is an IDE. It also has plugins, but most of the plugins that you will need to download to use in NetBeans or Eclipse are preconfigured. When you want to use some more advanced plugin, it may however be something you have to pay for, which should not be a problem when you are doing professional, paid work, should it? These things are not that expensive. For learning the subjects in this book, you will not need any plugin that is not in the community edition. As in this book, I will develop the samples using IntelliJ and I recommend that you follow me during your learning experience. I want to emphasize that the examples in this book are independent of the actual IDE to be used. You can follow the book using NetBeans, Eclipse, or even Emacs, notepad, or vi. IDE services Integrated development environments provide us with services. The most basic service is that you can edit files with them, but they also help build the code, find bugs, run the code, deploy to the application server in development mode, debug, and so on. In the following sections, we will look at these features. I will not give an exact and precise introduction on how to use one or the other IDE. A book like this is not a good medium for such a tutorial. IDEs differ on menu placement, keyboard shortcuts, and they may even change as newer versions are released. It is best to look at the actual IDE tutorial video or online help. Their features, on the other hand, are very similar. IntelliJ has the video documentation at https://www.jetbrains.com/idea/documentation/. IDE screen structure The different IDEs look similar, and have the same screen structure more or less. In the following screenshot, you can see an IntelliJ IDE: On the left side, you can see the file structure of a Java project. A Java project typically contains many files in different directories which we will discuss in the next chapter. The simple HelloWorld application contains a pom.xml project description file. This file is needed for the Maven build tool, which is also a topic for the next chapter. For now, you should only know that it is a file that describes the project structure for maven. The IDE also keeps track of some administrative data for itself. It is stored in HelloWorld.iml. The main program file is stored in the src/main/java directory and named HelloWorld.java. On the right side, you can see the files. In the screenshot, we have only one file opened. In case there is more than one file opened, then there are tabs-one for each file. Now, the active file is HelloWorld.java that can be edited in the source code editor. Editing files When editing, you can type in characters or delete characters, words, and lines, but this is something that all editors can do. IDEs offer extra. IDEs analyze the source code and format it, which, in turn, automatically indents the lines. It also continuously compiles the code in the background while you edit it and if there is some syntax error, then it underlines it with a red waiving line. When you fix the error, the red underlining disappears. The editor also automatically gives suggestions for further characters as you type. You can ignore the window that pops up and continue typing. However, many times, it is easier to stop after a character and use the up and down arrows to select the word that needs finishing before pressing Enter: the word will be inserted into the source code automatically. In the screenshot, you can see that I wrote System.o and the editor immediately suggested that I wanted to write out. The other alternatives are the other static fields and methods that are in the class System and which contain the letter o. The IDE editor gives you hints not only when it can type for you, but also when it cannot type instead of you. In the screenshot, the IDE tells you to type some expression as argument to the println()method that is boolean, char, int, and so on. The IDE has absolutely no idea what to type there. You have to construct the expression. Still, it can tell you that it needs to be of a certain type. It is not only the built-in types that the editor knows. The editor integrated with the JDK continuously scans the source files and knows what classes, methods, and fields are there in the source code which are usable at the place of editing. This knowledge is also heavily used when you want to rename a method or variable. The old method was to rename the field or method in the source file and then do an exhaustive search for all references to the variable. Using the IDE, the mechanical work is done by it. It knows all the uses of a field or method and automatically replaces the old identifier with the new one. It also recognizes whether a local variable happens to have the same name as the one that we rename, and the IDE only renames those occurrences that are really referring to the one we are renaming. You can usually do more than just renaming. There are more or less mechanical tasks that programmers call refactoring. These are supported by the IDEs using some keyboard shortcut and context sensitive menu in the editor—right click on the mouse and click Menu. The IDE also helps you to read the documentation of the libraries and source code as shown in the following image: Libraries provide Javadoc documentation for the public methods and you should also write Javadoc for your own method. Javadoc documentation is extracted from special comments in the source code and we will learn how to create those in Chapter 4, Mastermind - Creating a Game. These are located in comments in front of the actual method head. As creating compiled documentation is part of the compilation flow, the IDE also knows the documentation and it displays as a hovering box over the method names, class names, or whatever element you want to use in the source file when you position the cursor on the element. Managing projects On the left side of the IDE window, you can see the directory structure of the project. The IDE knows the different types of files and shows them in a way that is meaningful from the programming point of view. For example, it does not display Main.java as a filename. Instead, it displays Main and an icon that signals that Main is a class. It can also be an interface still in a file named Main.java but, in that case, the icon will show that this is an interface. This is again done by the IDE continuously scanning and compiling the code. The files are structured into subdirectories when we develop a Java code. These subdirectories follow the packaging structure of the code. Many times, in Java, we use compound and long package names, and displaying it as a deep and nested directory structure will not be so easy to handle. Packages are used to group the source files. The source files for classes that are related in some way should go into one package. We will discuss the notion of packages and how to use them in the next chapter The IDE is capable of showing the package structure instead of the nested directories for those directories of the project that contain source files. When you move a class or an interface from one package to another, it happens in a similar way as renaming or other refactoring. All references to the class or interface in the source files get renamed to the new package. If a file contains an import statement referring to the class, the name of the class in the statement is corrected. To move a class, you can open the package and use the good old drag and drop. Package hierarchy is not the only hierarchy displayed in the IDE. The classes are in packages but, at the same time, there is an inheritance hierarchy. Classes may implement interfaces and can extend other classes. The Java IDEs help us by showing type hierarchies where you can navigate across a graphical interface along the inheritance relations. There is another hierarchy that IDEs can show to help us with development: method call hierarchy. After analyzing the code, the IDE can show us the graph displaying the relations between the methods: which method calls which other methods. Sometimes, this call graph is also important in showing the dependencies of methods on each other. Build the code and run it The IDEs usually compile the code for analysis to help us spot syntax errors or undefined classes and methods on the fly. This compilation is usually partial, covering a part of the code, and as it runs all the time, the source code changes and is never actually complete. To create the deployable file, that is, the final deliverable code of the project, a separate build process has to be started. Most of the IDEs have some built-in tool for that, but it's not recommended to use these except for the smallest projects. Professional development projects use Ant, Maven, or Gradle instead. Here is an example of Maven. The IDEs are prepared to use such an external tool, and they can help us in starting them. This way, the build process can run on the developer machine without starting a new shell window. IDEs can also import the settings from the configuration file of these external build tools to recognize the project structure, where source files are, and what to compile to support the error checking while editing. The building process usually contains the execution of certain checks on the code. A bunch of the Java source file may compile smoothly and the code may still contain a lot of bugs and may be written in bad style, which will make the project becomes unmaintainable in the long run. To avoid such problems, we will use unit tests and static code analysis tools. These do not guarantee error free code but the chances are much better. IDEs have plugins to run the static code analysis tools as well as unit tests. Being integrated into the IDE has a huge advantage. When there is any problem identified by the analysis tool, or by some unit tests, the IDE provides an error message that also functions like a link on a web page. If you click on the message, usually blue and underlined, exactly like on a web page, the editor opens the problematic file and places the cursor where the issue is. Debugging Java Developing code needs debugging. Java has very good facilities to debug code during development. JVM supports debuggers via the Java Platform Debugger Architecture. This lets you execute code in debug mode and JVM will accept external debugger tools to attach to it via a network, or it will try to attach to a debugger depending on command-line options. JDK contains a client, the jdb tool, which contains a debugger; however, it is so cumbersome to use when compared to the graphical client built into the IDEs that I have never heard of anyone using it for real work. To start a Java program in debug mode so that JVM will accept a debugger client to attach the options to it, execute the following command: Xagentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=7896 The Xagentlib option instructs the Java runtime to load the jdwp agent. The part of the option that follows Xagentlib:jdwp= is interpreted by the debugger agent. These options are as follows: : This should specify which transport to use. It can be a shared memory (dt_shmem) socket or a TCP/IP socket transport but, in practice, you will always use the latter. This is specified in the preceding dt_socket sample. server: This specifies if the debugged JVM starts in server mode or client mode. When you start the JVM in server mode, it starts to listen on a socket and accepts the debugger to connect to it. If it is started in client mode, then it tries to connect a debugger that is supposed to be started in server mode, listening on a port. The value of the option is y meaning server mode or n meaning nonserver, a.k.a. client mode. suspend: This can also be y or n. If JVM is started in suspend mode, it will not start the Java code until a debugger is attached to it. If it is started with suspend=n, then the JVM starts and when a debugger attaches, it stops as soon as a breakpoint is reached. If you start a standalone Java application, you will usually start the debugging with suspend=y, which is the default. If you want to debug an application in an application server or servlet-container environment, then it is better to start with suspend=n; otherwise, the server does not start until the debugger attaches to it. Starting the Java process in suspend=y mode in case servlet application is only useful when you want to debug the servlet static initializer code, which is executed when the server is starting up. Without suspend mode, you will be required to attach the debugger very fast. It is better that JVM just waits for you in that situation. address: This should specify the address that JVM communicates with. If the JVM started in client mode, then it will start to connect to this address. If the JVM runs in server mode, then it will accept connections from the debugger on that address. The address may specify only the port. In this case, the IP address is that of the local machine. transport The other options the debugger agent may handle are for special cases. For the topics covered in this book, the preceding options are enough. The following screenshot shows a typical debugging session where we debug the simplest program in IntelliJ IDE: When you start a program from the IDE in debug mode, all these options are automatically set for you. You can set breakpoint just by clicking on the source code in the editor. You can have a separate form to add, remove, and edit breakpoints. Breakpoints can be attached to specific lines or specific events, like when an exception is thrown. Breakpoints attached to a specific line can also have conditions that tell the debugger to stop the execution of the code only when the condition is true; for example, if a variable has some predefined value. Summary In this chapter we were introduced to each other with Java. We do not know too much from each other but we got acquainted. We have installed the Java environment: Java, JDK and integrated development environment. We have written a small program and had a brief look at what can be done using the development tools. This is far from mastery but even the longest journey starts with a first step, which is sometimes the hardest to make. We have done it in our Java journey. We started rolling and for the enthusiasts that we are, nothing can stop us walking all the way long. The First Real Java Program - Sorting Names In the previous chapter, we got acquainted with Java, and especially with using the REPL tool and interactively executing some simple code. That is a good start, but we need more. In this chapter, we will develop a simple sort program. Using this code as an example, we will look at different build tools, which are frequently used for Java projects, and learn the basic features of the Java language. This chapter will cover the following topics: The sorting problem The project structure and build tools The Make, Ant, Maven, and Gradle build tools Java language features related to the code example Getting started with sorting The sorting problem is one of the oldest programming tasks that an engineer deals with. We have a set of records and we know that we want to find a specific one sometime later, and we want to find that one fast. To find it, we sort the records in a specific order that helps us find the record we want quickly. As an example, we have the names of students with their marks on some cards. When students come to the office asking for their results, we look through all of the cards one after the other to find the name of the enquiring student. However, it is better if we sort the cards by the names of the students alphabetically. When a student makes an enquiry, we can search the mark attached to the name much faster. We can look at the middle card; if it shows the name of the student, then we are happy to have found the name and the mark. If the card precedes the name of the student alphabetically, then we will continue searching in the second half; otherwise, we will check the first half. Following that approach, we can find the name of the student in a few steps. The number of steps can not be more than the number as many times the pack of cards can be halved. If we have two cards, then it is two steps at most. If it is four, then we will need three steps at most. If there are eight cards, then we may need four steps, but not more. If there are 1,000 cards, then we may need at most 11 steps, while the original, non-sorted set will need 1,000 steps, worst case. That is, approximately, it speeds up the search 100 times, so this is worth sorting the cards, unless the sorting itself takes too much time. The algorithm finding an element in the already sorted set we just described is called binary search (https://en.wikipedia.org/w iki/Binary_search_algorithm). In many cases, it is worth sorting the dataset, and there are many sorting algorithms to do that. There are simpler and more complex algorithms, and, as in many cases, more complex algorithms are the ones that run faster. As we are focusing on the Java programming part and not the algorithm forging, in this chapter, we will develop a Java code that implements a simple and not-that-fast algorithm. Bubble sort The algorithm that we will implement in this chapter is well-known as bubble sort. The approach is very simple. Begin at the start of the cards and compare the first and the second card. If the first card is later in lexicographic order than the second one, then swap the two cards. Then repeat this for the card that is at the second place now, then the third, and so on. There is a card that is lexicographically the latest, say Wilson. When we get this card and start to compare it with the next one, we will always swap them; this way, Wilson's card will travel to the last place where it has to be after the sort. All we have to do is repeat this travelling from the start and do the occasional swapping of cards again, but this time only to the last but one element. This time, the second latest element will get to its place—say, Wilkinson will be right before Wilson. If we have n cards, and we repeat this n-1 times, all cards will get to their place. In the following sections, we will create a Java project that implements this algorithm. Getting started with project structure and build tools When a project is more complex than a single class, and it usually is, then it is wise to define a project structure. We will have to decide where we store the source files, where the resource files (those that contain some resource for the program, but are not Java source) are, where the .class files should be written by the compiler, and so on. Generally, the structure is mainly the directory setup and the configuration of the tools that perform the build. The compilation of complex programs cannot be feasibly done using the command line issuing javac commands. If we have 100 Java source files, the compilation will require that many javac commands to be issued. It can be shortened using wild cards, such as javac *.java ,or we can write a simple bash script or a BAT command file that does that. First, it will be just 100 lines, each compiling one source Java file to class file. Then, we will realize that it is only time, CPU, and power consuming to compile the files that are not changed since the last compilations so we can add some bash programming that checks the time stamp on the source and generated files. Then, we will probably realize that... whatever. At the end, we will end up with a tool that is essentially a build tool. Build tools are available ready made; it is not worth reinventing the wheel. Instead of creating one, we will use a build tool that is ready. There are a few of them that can be found at https://en.wikipedia.org/wiki/List_of_build_automation_software. In this chapter, we will use one called Maven; however, before jumping into the details of this tool, we will look at some other tools that you are likely to meet as a Java professional in enterprise projects. In the following sections, we will discuss a bit of the four build tools: Make Ant Maven Gradle We will mention Make only briefly because it is not used in Java environments these days. However, Make was the first build tool, and many ideas that modern Java build tools are based on come from the good old make. You, as a professional Java developer, should also be familiar with Make so that you will not freak out if you happen to see the use of it in a project for some purpose, and can know what it is and where its detailed documentation can be found. Ant was the first build tool widely used for Java many years ago, and it is still used in many projects. Maven is newer than Ant, and it uses a different approach. We will look at it in detail. Maven is also the official build tool of the Apache software foundation for the Java project. We will also use Maven as a build tool in this chapter. Gradle is even newer, and it has started to catch up to Maven these days. We will use this tool in later chapters in more detail. Make The make program was originally created in April 1976, so this is not a new tool. It is included in the Unix system, so this tool is available without any extra installation on Linux, Mac OS X, or any other Unixbased system. Additionally, there are numerous ports of the tool on Windows, and some version is/was included in the Visual Studio compiler toolset. The Make is not tied to Java. It was created when the major programming language was C, but it is not tied to C or any other language. The make is a dependency description language that has a very simple syntax. The make, just like any other build tool, is controlled by a project description file. In the case of make, this file contains a rule set. The description file is usually named Makefile, but in case the name of the description file is different, it can be specified as a command-line option to the make command. Rules in Makefile follow each other and consist of one or more lines. The first line starts at the first position (there is no tab or space at the start of the line) and the following lines start with a tab character. Thus, Makefile may look something like the following code: run : hello.jar java -cp hello.jar HelloWorld hello.jar : HelloWorld.class jar -cf hello.jar HelloWorld.class HelloWorld.class : HelloWorld.java javac HelloWorld.java This file defines three so-called targets: run, hello.jar, and HelloWorld.class. To create HelloWorld.class, type the following line at the command prompt: make HelloWorld.class Make will look at the rule and see that it depends on HelloWorld.java. If the HelloWorld.class file does not exist, or HelloWorld.java is newer than the Java class file, make will execute the command that is written on the next line and it will compile the Java source file. If the class file was created following the last modification of HelloWorld.java, then make knows that there is no need to run the command. In the case of creating HelloWorld.class, the make program has an easy task. The source file was already there. If you issue the make hello.jar command, the procedure is more complex. The make command sees that in order to create hello.jar, it needs HelloWorld.class, which itself is also a target on another rule. Thus, it may need to be created. First, it starts the problem the same way as before. If HelloWorld.class is there, and is older than hello.jar, there is nothing to do. If it is not there, or is newer than hello.jar, then the jar -cf hello.jar HelloWorld.class command needs to be executed, although not necessarily at the moment when it realizes that it has to be performed. The make program remembers that this command has to be executed sometime in the future when all the commands that are needed to create HelloWorld.class are already executed successfully. Thus, it continues to create the class file exactly the same way as I described earlier. In general, a rule can have the following format: target : dependencies command The make command can create any target using the make target command by first calculating which commands to execute and then executing them one by one. The commands are shell commands executing in a different process and may pose problems under Windows, which may render the Makefile files' operating system dependent. Note that the run target is not an actual file that make creates. A target can be a file name or just a name for the target. In the latter case, make will never consider the target to be readily available. As we do not use make for a Java project, there is no reason to get into more details. Additionally, I cheated a bit by making the description of a rule simpler than it should be. The make tool has many powerful features out of the scope of this book. There are also several implementations that differ a little from each other. You will most probably meet the one made by the Free Software Foundation—the GNU make. And, of course, just in case of any Unix command-line tool, man is your friend. The man make command will display the documentation of the tool on the screen. The main points that you should remember about make are as follows: It defines the dependencies of the individual artifacts (targets) in a declarative way It defines the actions to create the missing artifacts in an imperative way This structure was invented decades ago and has survived up until now for most of the build tools, as you will see in the next few chapters. Ant The ant build tool was built especially for Java projects around the year 2000. The aim of Java to be a write-once-run-anywhere language needed a tool that can also be used in different environments. Although make is available on Unix machines, and Windows as well, Makefiles were not always compatible. There was a small problem with the use of the tab character that some editors replaced with space, rendering Makefile unusable, but this was not the major reason. The main problem with make that ignited the development of Ant is that the commands are shell commands. Even if the implementation of the make program was made to be compatible on different operating systems, the used commands were many times incompatible, and that was something make itself could not change. Because make issues external commands to build the targets, developers are free to use any external tool that is available for them on the development machine. Another machine using the same operating system just may not have the same set of tools invoked by make. This undermines the portability of the make built projects. At the same time, Ant is following the major principles of make. There are targets that may depend on each other and there are commands that need to be executed in an appropriate sequence to create the targets one after the other, following the dependency order. The description of the dependencies and the commands is XML (tab issue solved) and the commands are implemented in Java (system dependency is solved, well... more or less). As Ant is neither part of the operating system nor the JDK, you will have to download and install it separately if you want to use it. Installing Ant Ant can be downloaded from its official website (http://ant.apache.org). You can download the source or the precompiled version. The easiest way is to download the binary in a tar.gz format. Whenever you download software from the Internet, it is highly recommended that you check the integrity of the downloaded file. The HTTP protocol does not contain error checking, and it may happen that a network error remains hidden or a malevolent internal proxy modifies the downloaded file. Download sites usually provide checksums for the downloadable files. These are usually MD5, SHA1, SHA512, or some other checksums. When I downloaded the Apache Ant 1.9.7 version in tar.gz format, I also opened the page that led to the MD5 checksum. The checksum value is bc1d9e5fe73eee5c50b26ed411fb0119. The downloaded file can be checked using the following command line: $ md5 apache-ant-1.9.7-bin.tar.gz MD5 (apache-ant-1.9.7-bin.tar.gz) = bc1d9e5fe73eee5c50b26ed411fb0119 The calculated MD5 checksum is the same as the one on the website, which says that the file integrity is not harmed. On the Windows operating system, no tool to calculate MD5 digest is included. There is a tool that Microsoft provides, called File Integrity Checksum Verifier Utility, which is available via the page https://support.microsoft.com/en-us/help/841290/availability-and-description-of-the-fil e-checksum-integrity-verifier-utility. If you use Linux, it may happen that the md5 or md5sum utility is not installed. In that case, you can install it using the command apt-get or whatever installation tool your Linux distribution supports. After the file is downloaded, you can explode it to a subdirectory using the following command: tar xfz apache-ant-1.9.7-bin.tar.gz The created subdirectory is the usable binary distribution of Ant. Usually, I move it under ~/bin, making it available only for my user on OS X. After that, you should set the environment variable as ANT_HOME to point to this directory and also add the bin directory of the installation to the PATH. To do that, you should edit the ~/.bashrc file and add the following lines to it: export ANT_HOME=~/bin/apache-ant-1.9.7/ export PATH=${ANT_HOME}bin:$PATH Then, restart the terminal application, or just type . following command: ~/.bashrc and test the installation of Ant by typing the $ ant Buildfile: build.xml does not exist! Build failed If the installation was correct, you should see the preceding error message. Using Ant When you see a project to be built by Ant, you will see a build.xml file. This is the project build file, the one that Ant was missing when you checked that the installation was correct. It can have any other name, and you can specify the name of the file as a command-line option for Ant, but this is the default file name, as Makefile was for make. A build.xml sample looks like the following: <project name="HelloWorld" default="jar" basedir="."> <description> This is a sample HelloWorld project build file. </description> <property name="buildDir" value="build"/> <property name="srcDir" value="src"/> <property name="classesDir" value="${buildDir}/classes"/> <property name="jarDir" value="${buildDir}/jar"/> <target name="dirs"> <mkdir dir="${classesDir}"/> <mkdir dir="${jarDir}"/> </target> <target name="compile" depends="dirs"> <javac srcdir="${srcDir}" destdir="${classesDir}"/> </target> <target name="jar" depends="dirs,compile"> <jar destfile="${jarDir}/HelloWorld.jar" basedir="${classesDir}"/> </target> </project> The top-level XML tag is project. Each build file describes one project, hence the name. There are three possible attributes to the tag, which are as follows: : This defines the name of the project and is used by some IDEs to display it in the left panel identifying the project default: This names the target to use when no target is defined on the command line starting Ant basedir: This defines the initial directory used for any other directory name calculation in the build file name The build file can contain a description for the project, as well as properties in property tags. These properties can be used as variables in the attributes of the tasks between the ${ and } characters, and play an important role in the build process. The targets are defined in target XML tags. Each tag should have a name that uniquely identifies the target in the build file and may have a depends tag that specifies one or more other targets that this target depends on. In case there is more than one target, the targets are comma separated in the attribute. The tasks belonging to the targets are executed in the same order as the targets dependency chain requires, in a very similar way as we saw in the case of make. You can also add a description attribute to a target that is printed by Ant when the command-line option, projecthelp, is used. This helps the users of the build file to know what targets are there and which does what. Build files tend to grow large with many targets, and when you have ten or more targets, it is hard to remember each and every target. The sample project with HelloWorld.java is now arranged in the following directories: in the root folder of the project HelloWorld.java in the src folder of the project The build/ folder does not exist; it will be created during the build process The build/classes and build/jar also do not exist yet, and will be created during the build process build.xml When you start the build for the HelloWorld project the first time, you will see the following output: $ ant Buildfile: /Users/verhasp/Dropbox/java_9-by_Example/sources/ch02/build.xml dirs: [mkdir] Created dir: /Users/verhasp/Dropbox/java_9-by_Example/sources/ch02/build/classes [mkdir] Created dir: /Users/verhasp/Dropbox/java_9-by_Example/sources/ch02/build/jar compile: ... [javac] Compiling 1 source file to /Users/verhasp/Dropbox/java_9-by_Example/sources/ch02/build/classes jar: [jar] Building jar: /Users/verhasp/Dropbox/java_9-by_Example/sources/ch02/build/jar/HelloWorld.jar BUILD SUCCESSFUL Total time: 0 seconds Some unimportant lines are deleted from the actual output. Ant realizes that first it has to create the directories, then it has to compile the source code, and finally it can pack the .class files into a .jar file. Now it is up to you to remember the command to execute the HelloWorld application. It was listed already in the first chapter. Note that this time, the JAR file is named HelloWorld.jar, and it is not in the current directory. You can also try to read the online documentation of Ant and create a target run that executes the compiled and packed program. Ant has a built-in task named java that executes a Java class in almost the same way as you typed the java command in the terminal. Maven As Ant was created to overcome the shortages of make, Maven was created with a similar intention—to overcome the shortages of Ant. You may recall that make could not guarantee build portability because the commands make executes are arbitrary shell commands that may be system specific. An Ant build, if all the tasks are available on the classpath, is portable as long as Java runs the same way on the different platforms. The problem with Ant is a bit different. When you download the source code of a project and you want to build, what will the command be? You should ask Ant to list all the targets and select the one that seems to be the most suitable. The name of the task depends on the engineer who crafted the build.xml file. There are some conventions, but they are not strict rules. Where will you find the Java source files? Are they in the src directory or not? Will there also be some Groovy or other programming language files in case the project is polyglot? That depends. Again, there may be some conventions that some groups or company cultures suggest, but there is no general best industry practice. When you start a new project with Ant, you will have to create the targets for compilation, test execution, and packaging. It is something that you will have already done for other projects. After the second or third project, you will just copy and paste your previous build.xml to your new project. Is that a problem? Yes, it is. It is copy/paste programming, even if it is only some build files. Developers realized that a significant effort of the projects utilizing Ant is devoted to project build tool configuration, including repetitive tasks. When a new joiner comes to the team, they will first have to learn how the build is configured. If a new project is started, the build configuration has to be created. If it is a repetitive task, then better let the computers do it. That is generally what programming is all about, isn't it? Maven approaches the build issue a bit differently. We want to build Java projects. Sometimes, some Groovy or Jython things, but they are also JVM languages; thus, saying that we want to build Java projects is not really a huge restriction. Java projects contain Java files, sometimes some other programming language's source files, resource files, and generally, that is it. Ant can do anything, but we do not want to do just anything with a build tool. We want to build projects. Okay, after we restricted ourselves and accepted that we do not need a build tool that can be used for anything, we can go on. We can require that the source files be under the src directory. There are files that are needed for the operational code and there are files that contain some test code and data. Therefore, we will have two directories, src/test and src/main. Java files are in src/main/java as well as src/test/java. Resource files are under src/main/resources and src/test/resources. If you want to put your source files somewhere else, then don't. I mean it. It is possible, but I will not even tell you how. Nobody does it. I do not even have any idea why Maven makes it possible. Whenever you see a project that is using Maven as a build tool, the sources are organized like that. There is no need to understand the directory structure envisioned by the project's build engineer. It is always the same. How about the targets and the tasks? They are also the same for all Maven-based projects. What else would you like to do with a Java project other than compile, test, package, or deploy it? Maven defines these project life cycles for us. When you want to compile a project using Maven as a build tool, you will have to type $ mvn compile to compile the project. You can do that even before understanding what the project actually is. As we have the same directory structure and the same goals, the actual tasks leading to the goals are also all the same. When we create a Maven project, we do not have to describe what the build process has to do and how it has to do it. We will have to describe the project, and only the parts that are project specific. The build configuration of a Maven project is given in an XML file. The name of this file is usually pom.xml, and it should be in the root directory of the project, which should be the current working directory when firing up Maven. The word POM stands for Project Object Model, and it describes the projects in a hierarchical way. The source directories, the packaging, and other things are defined in a so-called super POM. This POM is part of the Maven program. Anything that the POM defines, overrides the defaults defined in the super POM. When there is a project with multiple modules, the POMs are arranged into a hierarchy, and they inherit the configuration values from the parent down to the modules. As we will use Maven to develop our sorting code, we will see some more details later. Installing Maven Maven is neither a part of the operating system nor the JDK. It has to be downloaded and installed in a very similar way to Ant. You can download Maven from its official website (https://maven.apache.org/) under the download section. Currently, the latest stable version is 3.3.9. When you download it, the actual release may be different; instead, use the latest stable version. You can download the source or the precompiled version. The easiest way is to download the binary in tar.gz format. I cannot skip drawing your attention to the importance of checking the download integrity using checksums. I have detailed the way to do it in the section about Ant installation. After the file is downloaded, you can explode it to a subdirectory using the following command: tar xfz apache-maven-3.3.9-bin.tar.gz The created subdirectory is the usable binary distribution of Maven. Usually, I move it under ~/bin, making it available only for my user on OS X. After that, you should add the bin directory of the installation to the PATH. To do that, you should edit the ~/.bashrc file and add the following lines to it: export M2_HOME=~/bin/apache-maven-3.3.9/ export PATH=${M2_HOME}bin:$PATH Then, restart the terminal application, or just type . follows: ~/.bashrc and test the installation of Maven typing, as $ mvn -v Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00) Maven home: /Users/verhasp/bin/apache-maven-3.3.9 Java version: 9-ea, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac" You should see a similar message on the screen that displays the installed Maven version and other information. Using Maven Unlike Ant, Maven helps you create the skeleton of a new project. To do that, you will have to type the following command: $ mvn archetype:generate Maven will first download the actually available project types from the network and prompt you to select the one you want to use. This approach seemed to be a good idea while Maven was new. When I first started Maven, the number of listed projects was somewhere between 10 and 20. Today, as I write this book, it lists 1,635 different archetypes. This number seems more like a historical date (the constitution of the French Academy of Science) than a usable size list of different archetypes. However, do not freak out. Maven offers a default value when it asks for your choice, and it is good for the HelloWorld we go for. Choose a number: 817: The actual number may be different on your installation. Whatever it is, accept the suggestion and press Enter. After that, Maven will ask you for the version of the project: Choose version: 1: 1.0-alpha-1 2: 1.0-alpha-2 3: 1.0-alpha-3 4: 1.0-alpha-4 5: 1.0 6: 1.1 Choose a number: 6: 5 Select the 1.0 version that is listed as number 5. The next thing Maven asks for is the group ID and the artifact ID of the project. The dependency management that we will discuss later uses these. I selected a group ID based on the book and the publisher. The artifact of the project is SortTutorial as we will start our chapter example in this project. Define value for property 'groupId': : packt.java9.by.example Define value for property 'artifactId': : SortTutorial The next question is the current version of the project. We have already selected 1.0 and Maven offers 1.0SNAPSHOT. Here, I selected 1.0.0-SNAPSHOT because I prefer semantic versioning. Define value for property 'version': 1.0-SNAPSHOT: : 1.0.0-SNAPSHOT Semantic versioning, defined on http://semver.org/, is a versioning scheme that suggests three digit version numbers as M.m.p. for Major, minor, and patch version numbers. This is very useful for libraries. You will increment the last version number if there is only a bug fix since the previous release. You will increment the minor number when the new release also contains new features, but the library is compatible with the previous version; in other words, any program that is using the older version can still use the newer version. The major release number is increased when the new version is significantly different from the previous one. In the case of application programs, there is no code that uses the application API; thus, the minor version number is not that important. It does not hurt, though, and it often proves to be useful to signal smaller changes in the application. We will discuss how to version software in the last chapter. Maven handles the versions that have the -SNAPSHOT postfix as non-release versions. While we develop the code, we will have many versions of our code, all having the same snapshot version number. On the other hand, non-snapshot version numbers can only be used only for a single version. Define value for property 'package': packt.java9.by.example: : The last question from the program skeleton generation is the name of the Java package. The default is the value we gave for groupId, and we will use this. It is a rare exception to use something else. When we have specified all the parameters that are needed, the final request is to confirm the setting: Confirm properties configuration: groupId: packt.java9.by.example artifactId: SortTutorial version: 1.0.0-SNAPSHOT package: packt.java9.by.example Y: : Y After entering Y, Maven will generate the files that are needed for the project and display the report about this: [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] [INFO] ----------------------------------------------------------Using following parameters for creating project from Old (1.x) Archetype: maven-archetype-quickstart:1.0 ----------------------------------------------------------Parameter: basedir, Value: .../mavenHelloWorld Parameter: package, Value: packt.java9.by.example Parameter: groupId, Value: packt.java9.by.example Parameter: artifactId, Value: SortTutorial Parameter: packageName, Value: packt.java9.by.example Parameter: version, Value: 1.0.0-SNAPSHOT *** End of debug info from resources from generated POM *** project created from Old (1.x) Archetype in dir: .../mavenHelloWorld/SortTutorial ----------------------------------------------------------BUILD SUCCESS ----------------------------------------------------------Total time: 01:27 min Finished at: 2016-07-24T14:22:36+02:00 Final Memory: 11M/153M ----------------------------------------------------------- You can take look at the following generated directory structure: You can also see that it generated the following three files: SortTutorial/pom.xml that contains the Project Object Model that contains a HelloWorld sample application SortTutorial/src/test/java/packt/java9/by/example/AppTest.java that contains a unit test skeleton utilizing the junit4 library SortTutorial/src/main/java/packt/java9/by/example/App.java We will discuss unit tests in the next chapter. For now, we will focus on the sorting application. As Maven was so kind and generated a sample class for the app, we can compile and run it without actual coding, just to see how we can build the project using Maven. Change the default directory to SortTutorial issuing cd SortTutorial and issue the following command: $ mvn package We will get the following output: Maven fires up, compiles, and packages the project automatically. If not, please read the next info box. When you first start Maven, it downloads a lot of dependencies from the central repository. These downloads take time, and are reported on the screen, so the actual output may be different from what you saw in the preceding code. Maven compiles code with the default settings for Java version 1.5. It means that the generated class file is compatible with Java version 1.5, and also that the compiler only accepts language constructs that were available already in Java 1.5. If we want to use newer language features, and in this book we use a lot, the pom.xml file should be edited to contain the following lines: <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.9</source> <target>1.9</target> </configuration> </plugin> </plugins> </build> When using Java 9's default settings for Maven, it becomes even more complex, because Java 9 does not generate class format nor restrict source compatibility earlier than Java 1.6. At this very moment, as I write these lines, the latest Maven release is 3.3.9. When I try to compile the preceding code without the modifications, the Java compiler stops with an error displaying the following: [ERROR] Source option 1.5 is no longer supported. Use 1.6 or later. [ERROR] Target option 1.5 is no longer supported. Use 1.6 or later. Later, Maven releases may behave differently in the future. Now, you can start the code using the following command: $ java -cp target/SortTutorial-1.0.0-SNAPSHOT.jar packt.java9.by.example.App You can see the result of a sample run in the following picture: Gradle Ant and Maven are two worlds, and using one or the other may lead to heated debates on Internet forums. Ant gives freedom to developers to create a build process that fits their taste. Maven restricts the team to use a build process that is more standard. Some special processes that do not match any standard build, but which are sometimes needed in some environments, are hard to implement using Maven. In Ant, you can script almost anything using the built-in tasks, almost the same way as you can program bash. Utilizing Maven is not that simple, and, it often requires writing a plugin. Even though writing a plugin is not rocket science, developers usually like to have the possibility of making things in a simpler way: Scripting. We have two approaches, two mindsets and styles, and not a single tool to fulfill all the needs. No surprise that by the Java technologies were developed, a new build tool was emerging. Gradle tries to use the best of both worlds, utilizing techniques that were not available by the time Maven and Ant were first developed. Gradle has built-in targets and life cycle, but at the same time, you can also write your own targets. You can configure a project, just like using Maven, without scripting the tasks to do so, but at the same time, you can also script your own target just like in Ant. What is more, Gradle integrated Ant, so any task implemented for Ant is available for Gradle as well. Maven and Ant use XML files to describe the build. Today, XML is a technology of the past. We still use it, and a developer should be fluent in handling, reading, and writing XML files, but a modern tool does not use XML for configuration. New, fancy formats such as JSON are more popular. Gradle is no exception. The configuration file of Gradle uses a domain-specific language (DSL) based on Groovy. This language is more readable for programmers and gives more freedom to program build processes. And, this is also the danger of Gradle. Having the powerful JVM language Groovy in the hands of developers to create build tools gives a freedom and temptation to create complex build processes that seem to be a good idea at the start, but later may prove to be just too complex and hard, and, therefore, expensive to maintain. This is exactly why Maven was implemented in the first place. I have to stop before getting into another area that is the ground for heated and pointless debates. Gradle is an extremely powerful build tool. You should use it carefully, just like you would use a weapon—don't shoot your legs. Installing Gradle To install Gradle, you will have to download the compiled binaries from the https://gradle.org/gradle-download/ website. Again, I'd like to emphasize the importance of checking the download integrity using checksums. I have given a detailed way to do it in the section about Ant installation. Unfortunately, the Gradle website does not provide the checksum values for the downloadable files. Gradle is downloadable in the ZIP format. To unpack the file, you will have to use the unzip command: $ unzip gradle-3.3-bin.zip The created subdirectory is the usable binary distribution of Gradle. Usually, I move it under ~/bin, making it available only for my user on OS X. After that, you should add the bin directory of the installation to the PATH. To do that, you should edit the ~/.bashrc file and add the following lines: export GRADLE_HOME=~/bin/gradle-3.3/ export PATH=${GRADLE_HOME}bin:$PATH Then, restart the terminal application, or just type . following: ~/.bashrc and test the installation of Gradle, typing the $ gradle -version We get to the following output, as can be seen in this screenshot: Setting up the project with Maven To start the project, we will use the directory structure and pom.xml that was created by Maven itself when we started with the following command line: $ mvn archetype:generate It created the directories, the pom.xml file, and an App.java file. Now, we will extend this project by creating new files. We will code the sorting algorithm first in the packt.java9.by.example.stringsort package: When we create the new package in the IDE, the editor will automatically create the stringsort subdirectory under the already existing src/main/java/packt/java9/by/example directory: Creating the new Sort class using the IDE will also automatically create a new file named Sort.java in this directory, and it will fill in the skeleton of the class: package packt.java9.by.example.stringsort; public class Sort { } We will now have App.java containing the following code: package packt.java9.by.example; public class App { public static void main( String[] args ) { System.out.println( "Hello World!" ); } } Maven created it as a starting version. We will edit this file to provide a sample list that the sorting algorithm can sort. I recommend that you use the IDE to edit the file and also to compile and run the code. The IDE provides a shortcut menu to start the code and this is a bit easier than typing the command in Terminal. Generally, it is recommended that you get acquainted with the IDE features to save time avoiding repetitive tasks, such as typing terminal commands. Professional developers use the command line almost exclusively to test command-line features and use the IDE whenever it is possible. Coding the sort Maven and the IDE created the files for the sort program. They form the skeleton for our code, and now it is time to grow some muscles on them to let it move. We spent quite some time to set up the project by visiting the different build tools, only to learn how to compile the code. I hope that this did not distract you much, but anyhow, we deserve to see some real code. First, we will create the code for the sorting code, and after that, the code that invokes the sorting. The code that invokes the sorting is a kind of testing code. For simplicity, we will now simply use a public static void main method to start the code. We will use the test framework in later chapters. As for now, the code for the sorting will look like this: package packt.java9.by.example.stringsort; public class Sort { public void sort(String[] names) { int n = names.length; while (n > 1) { for (int j = 0; j < n - 1; j++) { if (names[j].compareTo(names[j + 1]) > 0) { final String tmp = names[j + 1]; names[j + 1] = names[j]; names[j] = tmp; } } n--; } } } This is the class that does the sorting. There is only one method in this class that does the sorting. The argument to the method is an array containing the strings, and the method sorts this array. The method has no return value. This is denoted in the declaration using the pseudo type void. Methods use their arguments to perform some tasks, and may return one value. The arguments to the method are passed by value, which means that the method cannot modify the variable passed as argument. However, it can modify the objects the arguments contain. In this case, the array is modified and we will sort it. On the other hand, the actualNames variable will point to the same array and the sort method cannot do anything to make this variable point to a different array. There is no main method in this class, which means that it cannot be started from the command line on its own. This class can only be used from some other class, as every Java program should have a class that has a public static void main method that we created separately. I could also put a main method into the class to make it executable, but that is not a good practice. Real programs are composed of many classes, and one class should not do many things. Rather, it's the opposite. The single responsibility principle says that a single class should be responsible for one single thing; therefore, class sort does the sorting. Executing the application is a different task, and thus it has to be implemented in a different class. Often, we do not implement the class containing the main method. Often, a framework provides it. For example, writing a servlet that runs in a servlet container requires containing a class that implements the javax.servlet.Servlet interface. In this case, the program seemingly does not have a main method. The actual implementation of the servlet container does. The Java command line starts the container and the container loads the servlets when they are needed. In the following example code, we implemented the App class containing the main method: package packt.java9.by.example; import packt.java9.by.example.stringsort.Sort; public class App { public static void main(String[] args) { String[] actualNames = new String[]{ "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" }; final Sort sorter = new Sort(); sorter.sort(actualNames); for (final String name : actualNames) { System.out.println(name); } } } This code contains a string array initialized to contain constant values, creates a new instance of the Sort class, invokes the sort method, and then prints out the code to the standard output. In real programs, we almost never have such constants in program codes; we put them into resource files and have some code to read the actual values. This separates the code from data and eases maintenance, eliminating the risk of accidental modification of code structure when only the data is to be changed. Similarly, we will almost never write anything to standard output using System.out. Usually, we will use logging possibilities that are available from different sources. There are different libraries that provide logging functionalities and logging is also available from the JDK itself. As for now, we will focus on simple solutions so as to not distract your focus from Java by the plethora of different libraries and tools. In the following section, we will look at the Java language constructs that we used to code the algorithm. First, we will look at them generally, and then, in a bit more detail. These language features are not independent of each other: one builds up on the other, and therefore, the explanation will first be general, and we will go into details in the subsections. int n = names.length; <br/> while (n > 1) { <br/> for (int j = 0; j < n - 1; j++) { <br/> if (names[j].compareTo(names[j + 1]) > 0) { <br/> final String tmp = names[j + 1]; <br/> names[j + 1] = names[j]; <br/> names[j] = tmp; <br/> } <br/> } <br/> n--; <br/> } The n variable holds the length of the array at the start of the sorting. Arrays in Java always have a property that gives the length and it is called length. When we start the sorting, we will go from the start of the array to the end of it and, as you may recall, the last element, Wilson, will walk up to the last position during this first iteration. Subsequent iterations will be shorter and, therefore, the variable n will be decreased. Blocks The code in Java is created in code blocks. Anything that is between the { and } characters is a block. In the preceding example, the code of the method is a block. It contains commands, and some of them, like the while loop, also contain a block. Inside that block, there are two commands. One of them is a for loop, again with a block. Although we can have single expressions to form the body of a loop, we usually use blocks. We will discuss loops in detail in just a few pages. As we could see in the preceding example, the loops can be nested, and thus the { and } characters form pairs. A block can be inside another block, but two blocks cannot overlap. When the code contains a } character, it is closing the block that was opened last. Variables In Java, just like in almost any programming language, we use variables. The variables in Java are typed. It means that a variable can hold a value of a single type. It is not possible for a variable to hold an int type at some point in the program and later a String type. When variables are declared, their type is written in front of the variable name. Variables also have visibility scope. Local variables in methods can only be used inside the block in which they are defined. A variable can be used inside methods or they can belong to a class or an object. To differentiate the two, we usually call these variables fields. Types Each variable has one type. In Java, there are two major groups of type: primitive and reference types. The primitive types are predefined, and you cannot define or create a new primitive type. There are eight primitive types: byte, short, int, long, float, double, boolean, and char. The first four types, byte, short, int, and long, are signed numeric integer types, capable of storing positive and negative numbers on 8, 16, 32, and 64 bits. The float and double types store floating point numbers on 32 and 64 bits in the IEEE 754 floating-point format. The boolean type is a primitive type that can only be true or false. The char type is a character data type that stores a single 16-bit Unicode character. For each primitive type, there is a class that can store the same type of value. When a primitive type has to be converted to the matching class type it is done automatically. It is called auto boxing. These types are Byte, Short, Integer, Long, Float, Double, Boolean, and Character. Take, for example, the following variable declaration: Integer a = 113; This converts the value 113, which is an int number, to an Integer object. These types are part of the runtime, and also part of the language. Although there is no primitive counterpart of it, there is a very important and ubiquitous class that we have already used: String. A string contains characters. The major differences between primitive types and objects are that primitive types cannot be used to invoke methods, but they consume less memory. The difference between the memory consumption and its consequences for speed is important in the case of arrays. Arrays Variables can be a primitive type according to their declaration, or they may hold a reference to an object. A special object type is an array. When a variable holds a reference to an array, then it can be indexed with the [ and ] characters, along with an integral value consisting of 0 or a positive value ranging to one less than the array's length, to access a certain element of the array. Multi-dimensional arrays are also supported by Java when an array has elements that are also arrays. Arrays are indexed from zero in Java. Under or over indexing is checked at runtime, and the result is an exception. An exception is special condition that interrupts the normal execution flow and stops the execution of the code or jumps to the closest enclosing catch statement. We will discuss exceptions and how to handle them in the next chapter. When a code has an array of a primitive type, the array contains many memory slots, each holding the value of the type. When the array has a reference type, in other words, when it is an array of objects, then the array elements are references to objects, each containing the type. In the case of int for example, each element of the array is 32-bit, which is 4 bytes. If the array is a type of Integer, then the elements are references to objects, pointers, so to say, which is usually 64-bit using 64-bit JVM and 32-bit on 32-bit JVM. In addition to that, there is an Integer object somewhere in memory that contains the 4-byte value and also an object header that may be as much as 24 bytes. The actual size of the extra information needed to administer each object is not defined in the standard. It may be different on different implementations of the JVM. The actual coding, or even the optimization of the code in an environment, should not depend on the actual size. However, the developers should be aware that this overhead exists and is in the range of around 20 or so bytes for every object. Objects are expensive in terms of memory consumption. Memory consumption is one issue, but there is something else. When the program works with a large amount of data and the work needs the consecutive elements of the array, then the CPU loads a chunk of memory into the processor cache. It means that the CPU can access elements of the array that are consecutively faster. If the array is of a primitive type, it is fast. If the array is of some class type, then the CPU has to access memory to get the actual value, which may be as much as 50 times slower. Expressions Expressions in Java are very much like in other programming languages. You can use the operators that may be similar from languages such as C or C++. They are as follows: Unary prefix and postfix increment operators ( -- and ++ before and after a variable) Unary sign (+ and -) operators Logical (!) and bitwise (~) negation Multiplication (*), division (/), and modulo (%) Addition and subtraction (+ and - again, but this time as binary operators) Shift operators move the values bitwise, and there is left (<<) and right (>>) shift and unsigned right shift (>>>) The comparing operators are <, >, <=, >=, ==, != and instanceof that result in boolean value There are bitwise or (|), and (&), exclusive or (^) operators, and similarly logical or (||), and (&&) operators When logical operators are evaluated, they are shortcut evaluated. It means the right-hand operand is evaluated only if the result cannot be identified from the result of the left operand. The ternary operator is also similar to the one, like it is on C, selecting from one of the expressions based on some condition: condition ? expression 1 : expression 2. Usually, there is no problem with the ternary operator, but sometimes you have to be careful as there is a complex rule controlling the type conversions in case the two expressions are not of the same type. It's always better to have the two expressions be of the same type. Finally, there is an assignment operator (=) that assigns the value of an expression to a variable. For each binary operator, there is an assignment version that combines = with a binary operator to perform an operation involving the right operand and assign the result to the left operand, which must be a variable. These are +=, -=, *=, /=, %=, &=, ^=, |=, <<=, >>=, and >>>=. The operators have precedence and can be overridden by parentheses, as usual. An important part of expressions is invoking methods. Static methods can be invoked by the name of the class and the name of the method. For example, to calculate the sine of 1.22, we can write the following line: double z = Math.sin(1.22); Here, Math is the class from the package java.lang. The method sin is invoked without using any instance of Math. This method is static, and it is not likely that we will ever need any other implementation of it than the one provided in the class Math. Non-static methods can be invoked using an instance and the name of the method with a dot separating the two. For example, take the following code line as an example: System.out.println("Hello World"); The preceding code uses an instance of the class PrintStream that is readily available through a static field in the class System. This variable is called out, and when we write our code, we have to reference it as System.out. The method println is defined in the class PrintStream and we invoke it on the object referenced by the variable out. This example also shows that static fields can also be referenced through the name of the class and the field separated by a dot. Similarly, when we need to reference a non-static field, we can do it through an instance of the class. Static methods defined in the same class from where it is invoked or inherited can be invoked without the class name. Invoking a non-static method defined in the same class or being inherited can be invoked without an instance. In this case, the instance is the current object the execution is in. This object is also available through the this keyword. Similarly, when we use a field of the same class where our code is, we simply use the name. In case of a static field, the class we are in by default. In the case of a non-static field, the instance is the object referenced by the this keyword. You can also import a static method into your code using the importstatic language feature, in which case you can invoke the method without the name of the class. The arguments of the method calls are separated using commas. Methods and method argument passing is an important topic that we will mention in detail in a separate subsection. Loops The for loop inside the while loop will go through all the elements from the first (indexed with zero in Java) up till the last (indexed with n-1). Generally, the for loop has the same syntax as in C: for( initial expression ; condition ; increment expression ) block First, the initial expression is evaluated. It may contain variable declaration, as in our example. The variable j in the preceding example is visible only inside the block of the loop. After this, the condition is evaluated, and after each execution of the block, the increment expression is executed. The loop repeats so long as the condition is true. If the condition is false right after the execution of the initial expression, the loop does not execute at all. The block is a list of commands separated by semicolons and enclosed between the { and } characters. Instead of { and }, enclosed block Java lets you use a single command following the head of the for loop. The same is true in the case of the while loop, and also for the if...else constructs. Practice shows that this is not something a professional should use. Professional code always uses curly braces, even when there is only a single command where the block is in place. This prevents the dangling else problem and generally makes the code more readable. This is similar to many C-like languages. Most of them allow a single command at these places, and professional programmers avoid using a single command in these languages for readability purposes. It is ironic that the only language that strictly requires the use of the { and } braces at these places is Perl—the one language infamous for unreadable code. The loop in the for (int j = 0; j < n - 1; j++) { sample starts from zero and goes to n-2. Writing j < n-1 is the same, in this case, as j <= n-2. We will limit j to stop in the loop before the end of the section of the array, because we reach beyond the index j by one comparing and conditionally swapping the elements indexed by j and j+1. If we went one element further, we would try to access an element of the array that does not exist, and it would cause a runtime exception. Try and modify the loop condition to j < n or j <= n-1 and you will get the following error message: It is an important feature of Java that the runtime checks memory access and throws an exception in the case of bad array indexing. In the good old days, while coding in C, often, we faced unexplainable errors that stopped our code much later and at totally different code locations from where the real error was. Array index in C silently corrupted the memory. Java stops you as soon as you make a mistake. It follows the fail-fast approach that you also should use in your code. If something is wrong, the program should fail. No code should try to live with or overcome an error that comes from a coding error. Coding errors should be fixed before they cause even more damage. There are also two more loop constructs in Java: the while loop and the do loop. The example contains a while loop: it is the outer loop that runs so long as there are at least two elements that may need swapping in the array: while (n > 1) { The general syntax and semantics of the while loop is very simple, as seen here: while ( condition ) block Repeat the execution of the block so long as the condition is true. If the condition is not true at the very start of the loop, then do not execute the block at all. The do loop is also similar, but it checks the condition after each execution of the block: do block while( condition ); For some reason, programmers rarely use do loops. Conditional execution The heart of the sort is the condition and the value swapping inside the loop. if (names[j].compareTo(names[j + 1]) > 0) { final String tmp = names[j + 1]; names[j + 1] = names[j]; names[j] = tmp; } There is only one conditional command in Java, the if command. It has the following format: if( condition ) block else block The meaning of the code structure is quite straightforward. If the condition is true, then the first block is executed, otherwise, the second block is executed. The else keyword, along with the second block, is optional. If there is nothing to be executed in case that the condition is false, then there is no need for the else branch, just like in the example. If the array element indexed with j is later in the sort order than the element j+1, then we swap them, but if they are already in order, there is nothing to do with them. To swap the two array elements, we will use a temporary variable named tmp. The type of this variable is String, and this variable is declared to be final. The final keyword has different meanings depending on where it is used in Java. This may be confusing for beginners unless you are warned about it, just like now. A final class or method is a totally different thing than a final field, which is again different than a final local variable. Final variables In our case, tmp is a final local variable. The scope of this variable is limited to the block following the if statement, and inside this block, this variable gets a value only once. The block is executed many times during the code execution, and each time the variable gets into scope, it gets a value. However, this value cannot be changed in the block. This may be a bit confusing. You can think about it as having a new tmp each time the block executes. The variable gets declared and has an undefined value and can get a value only once. Final local variables do not need to get the value where they are declared. You can assign a value to a final variable some time later. It is important that there should not be a code execution that assigns a value to a final variable that was already assigned a value before. The compiler checks it and does not compile the code if there is a possibility of the reassignment of a final variable. To declare a variable to be final is generally to ease readability of the code. When you see a variable in a code declared to be final, you can assume that the value of the variable will not change and the meaning of the variable will always be the same wherever it was used in the method. It will also help you avoid some bugs when you try to modify some final variables and the IDE will immediately complain about it. In such situations, it is likely to be a programming mistake that is discovered extremely early. In principle, it is possible to write a program where all variables are final. It is generally a good practice to declare all final variables that can be declared to be final and, in case some variable may not be declared final, then try to find some way of coding the method a bit differently. If you need to introduce a new variable to do that, it probably means you were using one variable to store two different things. These things are of the same type and stored in the same variable at different times but, logically, they still are different things. Do not try to optimize the use of variables. Never use a variable because you already have a variable of the type in your code that is available. If it is logically a different thing, then declare a new variable. While coding, always prefer source code clarity and readability. In Java, especially, the Just In Time compiler will optimize all this for you. Although we do not explicitly tend to use the final keyword on the argument list of a method, it is good practice to make sure that your methods compile and work if the arguments are declared final. Some experts, including me, believe that the method parameters should have been made final by default in the language. This is something that will not happen in any version of Java, so long as Java follows the backward compatibility philosophy. Classes Now that we have looked at the actual code lines and have understood how the algorithm works, let's look at the more global structures of the code that brings it together: classes and packages enclosing the methods. Every file in a Java program defines a class. Any code in a Java program is inside a class. There is nothing like global variables or global functions as in C, Python, Go, or other languages. Java is totally object oriented. There can be more than one class in a single file, but usually one file is one class. Later, we will see that there are inner classes when a class is inside another class, but, for now, we will put one class into one file. There are some features in the Java language that we do not use. When the language was created, these features seemed to be a good idea. CPU, memory, and other resources, including mediocre developers, were also more limited than today. Some of the features, perhaps, made more sense because of these environmental constraints. Sometimes, I will mention these. In the case of classes, you can put more than one class into a single file so long as only one is public. That is bad practice, and we will never do that. Java never obsoletes these features. It is a philosophy of Java to remain compatible with all previous versions. This philosophy is good for the already written, huge amount of legacy code. Java code written and tested with an old version will work in a newer environment. At the same time, those features lure beginners to a wrong style. For this reason, sometimes, I will not even mention these features. For example, here, I could say: There is one class in a file. This would not be absolutely correct. At the same time, it is more or less pointless to explain in great detail a feature that I recommend not to be used. Later, I may simply skip them and "lie". There are not too many of those features. A class is defined using the class keyword and each class has to have a name. The name should be unique within the package (see the next section) and has to be the same as the name of the file. A class can implement an interface or extend another class, for which we will see an example later. A class can also be abstract, final, and public. These are defined with the appropriate keywords, as you will see in examples. Our program has two classes. Both of them are public. The public classes are accessible from anywhere. Classes that are not public are visible only inside the package. Inner and nested classes can also be private visible only inside the top-level class defined on the file level. Classes that contain a main method to be invoked by the Java environment should be public. That is because they are invoked by the JVM. The class starts at the beginning of the file right after the package declaration and everything between the { and } characters belong to the class. The methods, fields, inner or nested classes, and so on are part of the class. Generally, curly braces denote some block in Java. This was invented in the C language, and many languages follow this notation. Class declaration is some block, methods are defined using some block, loops, and conditional commands use blocks. When we use the classes, we will have to create instances of classes. These instances are objects. In other words, objects are created instantiating a class. To do that, the new keyword is used in Java. When the line final Sort sorter = new Sort(); is executed in the App class, it creates a new object instantiating the Sort class. We will also say that we created a new Sort object or that the type of the object is Sort. When a new object is created, a constructor of the object is invoked. A bit sloppy, I may say, that the constructor is a special method in the class that has the same name as the class itself and has no return value. That is because it returns the created object. To be precise, constructors are not methods. They are initializers and they do not return the new object. They work on the not-ready-yet object. When a constructor executing the object is not fully initialized, some of the final fields may not be initialized and the overall initialization still can fail if the constructor throws an exception. In our example, we do not have any constructor in the code. In such a case, Java creates a default constructor that accepts no argument and does not modify the already allocated but uninitialized object. If the Java code defines an initializer, then the Java compiler does not create a default one. A class can have many constructors, each having different parameter list. In addition to constructors Java classes can contain initializer blocks. They are blocks on the class level, the same level as the constructor and methods. The code in these blocks is compiled into the constructors and is executed when the constructor is executing. It is also possible to initialize static fields in static initializer blocks. These are the blocks on the top level inside the class with the static keyword in front of them. They are executed only once when the class is loaded. We named the classes in our example App and Sort. This is a convention in Java to name almost everything in CamelCase. CamelCase is when the words are written without spaces between them. The first word may start with lowercase or uppercase, and, to denote the start of the second and subsequent words, they start with uppercase. ForExampleThisIsALongCamelCase name. Class names start with an uppercase letter. This is not a requirement of the language formally, but this is a convention that every programmer should follow. These coding conventions help you create code that is easier to understand by other programmers, and lead to easier maintenance. Static code analyzer tools, such as Checkstyle (http://checkstyle.sourceforge.net/), also check that the programmers follow the conventions. Inner, nested, local, and anonymous classes I have already mentioned inner and nested classes in the previous section. Now we look at them in bit more detail. The details of inner and nested classes at this point may be difficult. Don't feel ashamed if you do not understand this section fully. If it is too difficult, skip to the next section and read about packages and return here later. Nested, inner, and local classes are rarely used, though they have their roles and use in Java. Anonymous classes were very popular in GUI programming with the Swing user interface that allowed developers to create Java GUI applications. With Java 8 and the lambda feature, anonymous classes are not so important these days, and with the emerging JavaScript and browser technology, the Java GUI became less popular. When a class is defined in a file on its own, it is called a top-level class. Classes that are inside another class are, obviously, not top-level classes. If they are defined inside a class on the same level as fields (variables that are not local to some method or other block), they are inner or nested classes. There are two differences between them. One is that nested classes have the static keyword before the class keyword at their definition, and inner classes don't. The other difference is that instances of nested classes can exist without an instance of the surrounding class. Inner class instances always have a reference to an instance of the surrounding class. Because inner class instances cannot exist without an instance of the surrounding class, their instance can only be created by providing an instance of the outer class. We will see no difference if the surrounding class instance is the actual this variable, but if we want to create an instance of an inner class from outside the surrounding class, then we have to provide an instance variable before the new keyword separated by a dot, just like if new were a method. For example, we could have a class named TopLevel that has a class named InnerClass, like in the following code snippet: public class TopLevel { class InnerClass { } } Then we can create an instance of the InnerClass from outside with only a TopLevel object, like in this snippet: TopLevel tl = new TopLevel(); InnerClass ic = tl.new InnerClass(); As inner classes have an implicit reference to an instance of the enclosing class, the code inside the inner class can access the fields and the methods of the enclosing class. Nested classes do not have an implicit reference to any instance of the enclosing class, and they may be instantiated with the new keyword without any reference to any instance of any other class. Because of that, they cannot access the fields of the enclosing class unless they are static fields. Local classes are classes that are defined inside a method, constructor, or an initializer block. We will soon talk about initializer blocks and constructors. Local classes can be used inside the block where they are defined. Anonymous classes are defined and instantiated in a single command. They are a short form of a nested, inner, or local class, and the instantiation of the class. Anonymous classes always implement an interface or extend a named class. The new keyword is followed by the name of the interface or the class with the argument list to the constructor between parentheses. The block that defines the body of the anonymous class stands immediately after the constructor call. In the case of extending an interface, the constructor can only be the one without argument. The anonymous class with no name cannot have its own constructors. In modern Java we usually use lambda instead of anonymous classes. Last but not least—well, actually, least I should mention that nested and inner classes can also be nested in deeper structures. Inner classes cannot contain nested classes, but nested classes can contain inner classes. Why? I have never met anyone who could reliably tell me the real reason. There is no architectural reason. It could be like that. Java does not permit that. However, it is not really interesting. If you happen to write code that has more than one level of class nesting then just stop doing it. Most probably you are doing something wrong. Packages Classes are organized into packages and the first code line in a file should specify the package that the class is in. package packt.java9.by.example.stringsort; If you do not specify the package, then the class will be in the default package. This should not be used, except in the simplest case when you want to try some code. With Java 9, you can use jshell for this purpose, so, as opposed to previous versions of Java, now the suggestion becomes very simple—never put any class in the default package. The name of the packages is hierarchical. The parts of the names are separated by dots. Using package names helps you avoid name collisions. Names of the classes are usually kept short and putting them into packages helps the organization of the program. The full name of a class includes the name of the package the class is in. Usually, we will put those classes into a package that are in some way related, and add something to a similar aspect of a program. For example, controllers in an MVC pattern program are kept in a single package. Packages also help you avoid name collision of classes. However, this only pushes the problem from class name collision to package name collision. We have to make sure that the name of the package is unique and does not cause any problem when our code is used together with any other library. When an application is developed, we just cannot know what other libraries will be used in later versions. To be prepared for the unexpected, the convention is to name the packages according to some Internet domain names. When a development company has the domain name acmecompany.com, then their software is usually under the com.acmecompany... packages. It is not a strict language requirement. It is only a convention to write the domain name from right to left, and use it as package name, but this proves to be fairly good in practice. Sometimes, like I do in this book, one can deviate from this practice so you can see that this rule is not carved in stone. When the rubber hits the road, and the code is compiled into byte code, the package becomes the name of the class. Thus, the full name of the Sort class is packt.java9.by.example.stringsort.Sort. When you use a class from another package, you can use this full name or import the class into your class. Again, this is on the language level. Using the fully qualified name or importing makes no difference when Java becomes byte code. Methods We have already discussed methods, but not in detail, and there are still some aspects that we should meet before we go on. There are two methods in the sample classes. There can be many methods in a class. Method names are also camel cased by convention, and the name starts with a lowercase letter, as opposed to classes. Methods may return a value. If a method returns a value, the method has to declare the type of the value it returns and, in that case, any execution of the code has to finish with a return statement. The return statement has an expression after the keyword, which is evaluated when the method is executed and is returned by the method. It is good practice to have only one single return from a method but, in some simple cases, breaking that coding convention may be forgiven. The compiler checks the possible method execution paths, and it is a compile-time error if some of the paths do not return a value. When a method does not return any value, it has to be declared to be void. This is a special type that means no value. Methods that are void, such as the public static void main method, may simply miss the return statement and just end. If there is a return statement, there is no place for any expression defining a return value after the return keyword. Again, this is a coding convention to not use the return statement in case of a method that does not return any value, but in some coding patterns, this may not be followed. Methods can be private, protected, public, and static, and we will discuss their meaning later. We have seen that the main method that was invoked when the program started is a static method. Such a method belongs to the class and can be invoked without having any instance of the class. Static methods are declared with the static modifier, and they cannot access any field or method that is not static. In our example, the sort method is not static, but as it does not access any field and does not call any nonstatic method (as a matter of fact, it does not call any method at all), it could just as well be static. If we change the declaration of the method to public static void sort(String[] names) { (note the word static), the program still works, but the IDE will give a warning while editing, for example: Static member 'packt.java9.by.example.stringsort.Sort.sort(java.lang.String[])' accessed via instance reference That is because you can access the method without an instance directly through the name of the Sort.sort(actualNames); class without the need of the sorter variable. Calling a static method via an instance variable is possible in Java (again something that seemed to be a good idea at the genesis of Java, but is probably not), but it may mislead the reader of the code into thinking that the method is an instance method. Making the sort method static, the main method can be as follows: public static void main(String[] args) { String[] actualNames = new String[]{ "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" }; Sort.sort(actualNames); for (final String name : actualNames) { System.out.println(name); } } It seems to be much simpler (it is), and, in case the method does not use any field, you may think that there is no reason to make a method non-static. During the first ten years of Java, static methods were in heavy use. There is even a term, utility class, which means a class that has only static methods and should not be instantiated. With the advent of Inversion of Control containers, we tend to use less static methods. When static methods are used, it is harder to use dependency injection, and it is also more difficult to create tests. We will discuss these advanced topics in the next few chapters. For now, you are informed as to what static methods are and that they can be used; however, usually, unless there is a very special need for them, we will avoid them. Later, we will look at how classes are implemented in the hierarchy, and how classes may implement interfaces and extend other classes. When these features are looked at, we will see that there are socalled abstract classes that may contain abstract methods. These methods have the abstract modifier, and they are not defined—only the name, argument types (and names), and return type are specified. A concrete (non-abstract) class extending the abstract class should define them. The opposite of abstract method is the final method declared with the final modifier. A final method cannot be overridden in subclasses. Interfaces Methods are also declared in interfaces. A method declared in an interface does not define the actual behavior of the method; they do not contain the code. They have only the head of the method; in other words, they are abstract implicitly. Although nobody does, you may even use the abstract keyword in an interface when you define a method. Interfaces look very similar to classes, but instead of using the class keyword, we use the interface keyword. Because interfaces are mainly used to define methods, the methods are public if no modifier is used. Interfaces can also define fields, but since interfaces cannot have instances (only implementing classes can have instances), these fields are all static and they also have to be final. This is the default for fields in interfaces, thus we do not need to write these if we defined fields in interfaces. It was a common practice to define only constants in some interfaces and then use these in classes. To do that, the easiest way was to implement the interface. Since these interfaces do not define any method, the implementation is nothing more than writing the implements keyword and the name of the interface into the header of the class declaration. This is bad practice because this way the interface becomes part of the public declaration of the class, although these constants are needed inside the class. If you need to define constants that are not local to a class but are used in many classes, then define them in a class and import the fields using import static or just use the name of the class and the field. Interfaces can also have nested classes, but they cannot have inner classes. The obvious reason for that is that inner class instances have a reference to an instance of the enclosing class. In the case of an interface, there are no instances, so an inner class could not have a reference to an instance of an enclosing interface, because that just does not exist. The joyful part of it is that we do not need to use the static keyword in the case of nested classes because that is the default, just as in the case of fields. With the advent of Java 8, you can also have default methods in interfaces that provide default implementation of the method for the classes that implement the interface. There can also be static and private methods in interfaces since Java 9. Methods are identified by their name and the argument list. You can reuse a name for a method and have different argument types; Java will identify which method to use based on the types of the actual arguments. This is called method overloading. Usually, it is easy to tell which method you call, but when there are types that extend each other, the situation becomes more complex. The standard defines very precise rules for the actual selection of the method that the compiler follows, so there is no ambiguity. However, fellow programmers who read the code may misinterpret overloaded methods or, at least, will have hard time identifying which method is actually called. Method overloading may also hinder backward compatibility when you want to extend your class. The general advice is to think twice before creating overloaded methods. They are lucrative, but may sometimes be costly. Argument passing In Java, arguments are passed by value. When the method modifies an argument variable, then only the copy of the original value is modified. Any primitive value is copied during the method call. When an object is passed as an argument, then the copy of the reference to the object is passed. That way, the object is available to be modified for the method. In the case of classes that have their primitive counterpart, and also in the case of String and some other class types, the objects simply do not provide methods or fields to modify the state. This is important for the integrity of the language, and to not get into trouble when objects and primitive values automatically get converted. In other cases, when the object is modifiable, the method can effectively work on the very object it was passed to. This is also the way the sort method in our example works on the array. The same array, which is also an object itself, is modified. This argument passing is much simpler than it is in other languages. Other languages let the developer mix the pass by reference and the pass by value argument passing. In Java, when you use a variable by itself as an expression to pass a parameter to a method, you can be sure that the variable itself is never modified. The object it refers to, however, in case it is mutable, may be modified. An object is mutable if it can be modified, altering the value of some of its field directly or via some method call. When a class is designed in a way that there is no normal way to modify the state of the object after the creation of the object, the object is immutable. The classes Byte, Short, Integer, Long, Float, Double, Boolean, Character, as well as String, are designed in the JDK so that the objects are immutable. It is possible to overcome the limitation of immutability implementation of certain classes using reflection, but doing that is hacking and not professional coding. Doing that can be done for one single purpose—getting a better knowledge and understanding of the inner workings of some Java classes, but nothing else. Fields Fields are variables on the class level. They represent the state of an object. They are variables, with defined type and possible initial value. Fields can be static, final, transient, and volatile, and the access may be modified with the public, protected, and private keywords. Static fields belong to the class. It means that there is one of them shared by all the instances of the class. Normal, non-static fields belong to the objects. If you have a field named f, then each instance of the class has its own f. If f is declared static, then the instances will share the very same f field. The final fields cannot be modified after they are initialized. Initialization can be done on the line where they are declared, in an initializer block or in the constructor code. The strict requirement is that the initialization has to happen before the constructor returns. This way, the meaning of the final keyword is very different, in this case, from what it means in the case of a class or a method. A final class cannot be extended and a final method cannot be overridden in an extending class, as we will see in the next chapter. The final fields are either uninitialized or get a value during instance creation. The compiler also checks that the code does initialize all final fields during the object-instance creation or during the class loading, in case the final field is static, and that the code is not accessing/reading any final field that was not yet initialized. It is a common misconception that the final fields have to be initialized at the declaration. It can be done in an initializer code or in a constructor. The restriction is that, no matter which constructor is called in case there are more, the final fields have to be initialized exactly once. The transient fields are not part of the serialized state of the object. Serialization is an act of converting the actual value of an object to physical bytes. Deserialization is the opposite when the object is created from the bytes. It is used to save the state in some frameworks. The code that does the serialization, java.lang.io.ObjectOutputStream, works only with classes that implement the Serializable interface, and uses only the fields from those objects that are not transient. Very obviously, transient fields are also not restored from the bytes that represent the serialized form of the object because their value is not there. Serialization is usually used in distributed programs. A good example is the session object of a servlet. When the servlet container runs on a clustered node, some fields of objects stored into the session object may magically disappear between HTTP hits. That is because serialization saves and reloads the session to move the session between the nodes. Serialization, in such a situation, may also be a performance issue if a developer does not know the side effects of the stored large objects in the session. The volatile keyword is a keyword that tells the compiler that the field may be used by different threads. When a volatile field is accessed by any code, the JIT compiler generates code which ensures that the value of the field accessed is up to date. When a field is not volatile, the compiler-generated code may store the value of the field in a processor cache or registry for faster access when it sees that the value will be needed soon by some subsequent code fragment. In the case of volatile fields, this optimization cannot be done. Additionally, note that saving the value to memory and loading from there all the time may be 50 or more times slower than accessing a value from a registry or cache. Modifiers Methods, constructors, fields, interfaces, and classes can have access modifiers. The general rule is that in case there is no modifier, the scope of the method, constructor, and so on, is the package. Any code in the same package can access it. When the private modifier is used, the scope is restricted to the so-called compilation unit. This means the class that is in one file. What is inside one file can see and use anything declared to be private. This way, inner and nested classes can have access to each other's private variables, which may not really be a good programming style, but Java permits that. The opposite of private is public. It extends the visibility to the whole Java program, or at least to the whole module, if the project is a Java 9 module. There is a middle way: protected. Anything with this modifier is accessible inside the package and also in classes that extend the class (regardless of package) that the protected method, field, and so on, is in. Object initializers and constructors When an object is instantiated, the appropriate constructor is called. The constructor declaration looks like a method with the following deviation: the constructor does not have a return value. That is because the constructors work on the not-fully-ready instance when the new command operator is invoked and does not return anything. Constructors, having the same name as the class, cannot be distinguished from each other. If there is a need for more than one constructor, they have to be overloaded. Constructors, thus, can call each other, almost as if they were void methods with different arguments. However, there is a restriction—when a constructor calls another, it has to be the very first instruction in the constructor. You use this() syntax with an appropriate argument list, which may be empty, to invoke a constructor from another constructor. The initialization of the object instance also executes initializer blocks. These are blocks containing executable code inside the { and } characters outside the methods and constructors. They are executed before the constructor in the order they appear in the code, together with the initialization of the fields in case their declarations contain value initialization. If you see the static keyword in front of an initializer block, the block belongs to the class and is executed when the class is loaded along with the static field initializers. Compiling and running the program Finally, we will compile and execute our program from the command line. There is nothing new in this one; we will only apply what we have learned in this chapter using the following two commands: $ mvn package This compiles the program, packages the result into a JAR file, and finally executes the following command: $ java -cp target/SortTutorial-1.0.0-SNAPSHOT.jar packt.java9.by.example.App This will print the following result on the command line: Summary In this chapter, we have developed a very basic sort algorithm. It was made purposefully simple so that we could reiterate the basic and most important Java language elements, classes, packages, variables, methods, and so on. We also looked at build tools, so we are not empty handed in the next chapters when projects will contain more than just two files. We will use Maven and Gradle in the following chapters. In the very next chapter, we will make the sort program more complex, implementing more effective algorithms and also making our code flexible, giving us the opportunity to learn more advanced Java language features. Optimizing the Sort - Making Code Professional In this chapter, we will develop the sorting code and make it more general. We want to sort not only an array of Strings. Essentially, we will write a program that can sort anything that is sortable. That way, we will bring the coding to its full extent toward one of the major strengths of Java: abstraction. Abstraction, however, does not come without a price tag. When you have a class that sorts strings and you accidentally mix an integer or something else, which is not a string, into the sortable data, then the compiler will complain about it: Java does not allow you to put an int into a String array. When the code is more abstract, such programming errors may slip in. We will look at how to handle such exceptional cases catching and throwing Exceptions. To identify the bugs, we will use unit testing, applying the industry standard JUnit version 4. As JUnit heavily uses annotation, and because annotations are important, you will learn about it a bit. After that, we will modify the code to use the generics feature of Java that was introduced into the language in version 5. Using that possibility, we will catch the coding error during compilation time, which is better than during run time. The earlier a bug is identified, the cheaper it is to fix. For the build, we will still use Maven, but this time, we will split the code into small modules. Thus, we will have a multi-module project. We will have separate modules for the definition of a sorting module and for the different implementations. That way, we will look at how classes can extend each other and implement interfaces, and, generally, we will really start to program in the object-oriented way. We will also discuss Test Driven Development (TDD), and at the end of the section, we will start using the brand new feature of Java 9: module support. In this chapter, we will cover the following topics: Object-oriented programming principles Unit testing practices Algorithmic complexity and quick sort Exception handling Recursive methods Module support The general sorting program In the previous chapter, we implemented a simple sort algorithm. The code can sort elements of a String array. We did this to learn. For practical use, there is a ready cooked sort solution in the JDK that can sort members of collections, which are comparable. The JDK contains a utility class called Collections. This class contains a static Collections.sort method that is capable of sorting any List that has members that are Comparable. List and Comparable are interfaces defined in the JDK. Thus, if we want to sort a list of Strings, the simplest solution is as follows: public class SimplestStringListSortTest { @Test public void canSortStrings() { ArrayList actualNames = new ArrayList(Arrays.asList( "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" )); Collections.sort(actualNames); Assert.assertEquals(new ArrayList<String>(Arrays.<String>asList( "Abraham", "Dagobert", "Johnson", "Wilkinson", "Wilson")), actualNames); } } This code fragment is from a sample JUnit test, which is the reason we have the @Test annotation in front of the method. We will discuss that in detail later. To execute that test, you can issue the following command: $ mvn -Dtest=SimplestStringListSortTest test This sort implementation, however, does not fit our needs. First of all, because it is there ready (no need to code) and using it does not need anything new that you have not learned in the previous chapters. Except for the annotation in front of the method, there is nothing new in the code that you cannot understand. You may refresh BY turning some pages back, or else consult the oracle online documentation of the JDK (https://docs.oracle.com/javase/8/docs/api/), but that is all. You already know these things. You may wonder why I wrote the URL for the Java version 8 API to the link. Well, then this is the moment of honesty and truth-when I wrote this book, the Java 9 JDK was not available in its final form. I created most of the examples on my Mac Book using Java 8 and I only tested the features that are Java 9 specific. Support at the moment for Java 9 in the IDEs is not perfect. When you read this book, Java 9 will be available, so you can try and change that one single digit from 8 to 9 in the URL and get to the documentation of the version 9. At the moment, I get HTTP ERROR 404. Sometimes, you may need the documentation of older versions. You can use 3, 4, 5, 6, or 7 instead of 8 in the URL. Documentation for 3 and 4 is not available to read online, but it can be downloaded. Hopefully, you will never need that anymore. Version 5, perhaps. Version 6 is still widely used at large corporations. Although you can learn a lot from reading code that was written by other programmers, I do not recommend trying to learn from the JDK source code at this early stage of your studies. These blocks of code are heavily optimized, not meant to be tutorial codes, and old. They do not get rusted during the years, but they were not refactored to follow the coding styles of Java as it matured. At some places, you can find really ugly code in the JDK. Okay, saying that we need to develop a new sort code because we can learn from it, is a bit contrived. The real reason why we need a sort implementation is that we want something that can sort not only List data types and a List of something that implements the Comparable interface. We want to sort a bunch of objects. All we require is that the bunch containing the objects provides simple methods that are just enough to sort them and have a sorted bunch. Originally I wanted to use the word collection instead of bunch, but there is a Collection interface in Java and I wanted to emphasize that we are not talking about java.util.Collection of objects. We also do not want the objects to implement the Comparable interface. If we require the object to implement the Comparable interface, it may violate the Single Responsibility Principle (SRP). When we design a class, it should model some object class of the real world. We will model the problem space with classes. The class should implement the features that represent the behavior of the objects that it models. If we look at the example of students from the second chapter, then a Student class should represent the features that all students share, and is important from the modeling point of view. A Student object should be able to tell the name of the student, the age, the average scores of the last year, and so on. All students have feet, and certainly each of those feet have size so we may think that a Student class should also implement a method that returns the size of the student's foot (one for the left and one for the right just to be precise), but we do not. We do not, because the size of the foot is irrelevant from the model point of view. If we want to sort a list containing Student objects, the Student class has to implement the Comparable interface. But wait! How do you compare two students? By names, by age. or by the average score of them? Comparing a student to another is not a feature of the Student. Every class, or for that matter, package, library, or programming unit should have one responsibility and it should implement only that and nothing else. It is not exact. This is not mathematics. Sometimes, it is hard to tell if a feature fits into the responsibility or not. There are simple techniques. For example, in case of a student, you can ask the real person about his name and age, and probably they can also tell you their average score. If you ask one of them to compareTo (another student), as the Comparable interface requires this method, they will probably ask back, but by what attribute? Or how? Or just, what? In such a case, you can suspect that implementing the feature is probably not in the area of that class and this concern; the comparison should be segregated from the implementation of the original class. This is also called Segregation of Concerns, which is closely related to SRP. JDK developers were aware of this. Collections.sort that sorts a List of Comparable elements is not the only sorting method in this class. There is another that just sorts any List if you pass a second argument and object that implements the Comparator interface and is capable of comparing two elements of List. This is a clean pattern to separate the concerns. In some cases, separating the comparison is not needed. In other cases, it is desirable. The Comparator interface declares one single method that the implementing classes have to provide: compare. If the two arguments are equal, then this method returns 0. If they are different, it should return a negative or a positive int depending on which argument precedes which one. There are also sort methods in the JDK class, java.util.Arrays. They sort arrays, or only a slice of an array. The method is a good example of method overloading. There are methods with the same name, but with different arguments to sort a whole array for each primitive type, for a slice of each, and also two for object array implementing the Comparable interface, and also for object array to be sorted using Comparator. As you see, there is a whole range of sort implementations available in the JDK, and in 99 percent of the cases, you will not need to implement a sort yourself. The sorts use the same algorithm, a stable merge sort with some optimization. What we will implement is a general approach that can be used to sort lists, arrays, or just anything that has elements and it is possible to swap any two elements of it; the solution will be able to use the bubble sort that we have already developed and also other algorithms. A brief overview of various sorting algorithms There are many different sorting algorithms. As I said, there are simpler and more complex algorithms and, in many cases, more complex algorithms are the ones that run faster. In this chapter, we will implement the bubble sort and quick sort. We have already implemented the bubble sort for strings in the previous chapter, so in this case, the implementation will mainly focus on the recoding for general sortable object sorting. Implementing quick sort will involve a bit of algorithmic interest. Be warned that this section is here to give you only a taste of algorithmic complexity. It is far from precise and I am in the vain hope that no mathematician reads this and puts a curse on me. Some of the explanations are vague. If you want to learn computer science in depth, then after reading this book, find some other books or visit online courses. When we talk about the general sorting problem, we will think about some general set of objects that can be compared and any two of them can be swapped while we sort. We will also assume that this is an inplace sort; thus, we do not create another list or array to collect the original objects in sorted order. When we talk about the speed of an algorithm, we are talking about some abstract thing and not milliseconds. When we want to talk about milliseconds, actual real-world duration, we should already have some implementation in some programming language running on a real computer. Algorithms, in their abstract form, don't do that without implementation. Still, it is worth talking about the time and memory need of an algorithm. When we do that, we will usually investigate how the algorithm behaves for a large set of data. For a small set of data, most algorithms are just fast. Sorting two numbers is usually not an issue, is it? In case of sorting, we will usually examine how many comparisons are needed to sort a collection of n elements. Bubble sort needs approximately n2 (n times n) comparisons. We cannot say that this is exactly n2 because in case of n=2, the result is 1, for n=3 it is 3, for n=4 it is 6, and so on. However, as n starts to get larger, the actual number of comparisons needed and n2 will asymptotically be of the same value. We say that the algorithmic complexity of the bubble sort is O(n2). This is also called the big-O notation. If you have an algorithm that is O(n2) and it works just fine for 1,000 elements finishing in a second, then you should expect the same algorithm finishing for 1 million elements in around ten days or in a month. If the algorithm is linear, say O(n), then finishing 1,000 element in one second should make you expect 1 million to be finished in 1,000 seconds. That is a bit longer than a coffee break and too short for lunch. This makes it feasible that if we want some serious business sorting objects, we will need something better than bubble sort. That many unnecessary comparisons are not only wasting our time, but also CPU power, consuming energy, and polluting the environment. The question, however, is: how fast can a sort be? Is there a provable minimum that we cannot overcome? The answer is yes. When we implement any sorting algorithm, the implementation will execute comparisons and element swaps. That is the only way to sort a collection of objects. The outcome of a comparison can have two values. Say, these values are 0 or 1. This is one bit of information. If the result of the comparison is 1, then we swap, if the result is 0, then we do not swap. We can have the objects in different orders before we start the comparison and the number of different orders is n! (n factorial). That is, the numbers multiplied from 1 to n, in other words n!=1*2*3*...*(n1)*n. Let's assume that we stored the result of the individual comparisons in a number as a series of bits for each possible input for the sort. Now, if we reverse the execution of the sort and run the algorithm starting from the sorted collection, control the swapping using the bits that described the results of the comparison, and we use the bits the other way around doing the last swap first and the one that was done first during the sorting first, we should get back the original order of the objects. This way, each original order is uniquely tied to a number expressed as an array of bits. Now, we can express the original question this way: how many bits are needed to describe n factorial different numbers? That is exactly the number of comparisons we will need to sort n elements. The number of bits is log2(n!) . Using some mathematics, we will know that log2(n!) is the same as log2(1)+ log2(2)+...+ log2(n). If we look at this expression's asymptotic value, then we can say that this is the same O(n*log n). We should not expect any general sorting algorithm to be faster. For special cases, there are faster algorithms. For example, if we want to sort 1 million numbers that are each between one and 10, then we only need to count the number of the different numbers and then create a collection that contains that many ones, twos, and so on. This is an O(n) algorithm, but this is not applicable for the general case. Again, this was not a formal mathematical proof. Quick sort Sir Charles Antony Richard Hoare developed the quick sort algorithm in 1959. It is a typical divide and conquer algorithm. To sort a long array, pick an element from the array that will be the pivot element. Then, partition the array so that the left side will contain all the elements that are smaller than the pivot and the right side will contain all the elements that are larger than, or equal to the pivot. When this is done, the left side and the right side of the array can be sorted by calling the sort recursively. To stop the recursion, when we have one single element in the array, we will declare it sorted. We talk about a recursive algorithm when the algorithm is defined partially using itself. The most famous recursive definition is the Fibonacci series that is 0 and 1 for the first two elements and any later element the nth element is the sum of the (n-1)th and the (n-2)th element. Recursive algorithms are many times implemented in modern programming languages implementing a method that does some calculation but sometimes calls itself. When designing recursive algorithms, it is of utmost importance to have something that stops the recursive calls; otherwise, recursive implementation will allocate all memory available for the program stack and stop the program with error. The partitioning algorithm goes the following way: we will start to read the array using two indices from the start and end. We will first start with the index that is small and increase the index until it is smaller than the large index, or until we find an element that is greater than or equal to the pivot. After this, we will start to decrease the larger index so long as it is greater than the small index and the element indexed is greater than or equal to the pivot. When we stop, we swap the two elements pointed by the two indices, if the indices are not the same, and we will start increasing and decreasing the small and large indices, respectively. If the indices are the same, then we are finished with the partitioning. The left side of the array is from the start to the index where the indices met minus one; the right side starts with the index and lasts until the end of the to-be-sorted array. This algorithm is usually O(n log n), but in some cases it can degrade to be O(n2), depending on how the pivot is chosen. There are different approaches for the selection of the pivot. In this book, we will use the simplest: we will select the first element of the sortable collection as a pivot. <project> <br/>... <br/> <modules> <br/> <module>SortInterface</module> <br/> <module>bubble</module> <br/> <module>quick</module> <br/> </modules> <br/> </project> $ tree <br/> |-SortInterface <br/> |---src/main/java/packt/java9/by/example/ch03 <br/> |bubble <br/> |---src <br/> |-----main/java/packt/java9/by/example/ch03/bubble <br/> |----test/java/packt/java9/by/example/ch03/bubble <br/> |-quick/src/ <br/> |-----main/java <br/> |-----test/java Maven dependency management Dependencies are also important in the POM file. The previous project did not have any dependency, but this time we will use JUnit. Dependencies are defined in pom.xml using the dependencies tag. For example, the bubble sort module contains the following piece of code: <dependencies> <dependency> <groupId>packt.java9.by.example</groupId> <artifactId>SortInterface</artifactId> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> </dependency> </dependencies> The actual pom.xml in the code set you can download will contain more code than this. In print, we often present a version or only a fraction that helps the understanding of the topic that we are discussing at that point. It tells Maven that the module code uses classes, interfaces, and enum types that are defined in these modules that are available from some repository. When you use Maven to compile the code, the libraries that are used by your code are available from repositories. When Ant was developed, the notion of repositories was not invented. At that time, the developers copied the used version of the library into a folder in the source code structure. Usually, the directory lib was used for the purpose. There were two problems with this approach. One is the size of the source code repository. If, for example, 100 different projects used JUnit, then the JAR file of the JUnit library was copied there 100 times. The other problem was to gather all the libraries. When a library used another library, the developers had to read the documentation of the library that described (many times outdated and not precise) what other libraries are needed to use this library. Those libraries had to be downloaded and installed the same way. This was time consuming and error prone. When a library was missing and the developers just did not notice it, the error was manifested during compile time when the compiler could not find the class or even only at runtime when the JVM was not able to load the class. To solve this issue, Maven comes with a built-in repository manager client. The repository is a storage that contains the libraries. As there can be other types of files in a repository, not only libraries, Maven terminology is artifact. The groupId, the artifactId, and the version number identify an artifact. There is a very strict requirement that an artifact can only be put into a repository once. Even if there is an error during the release process that is identified after the erroneous release was uploaded, the artifact cannot be overwritten. For the same groupId, artifactId, and version, there can only be one single file that will never change. If there was an error, then a new artifact is to be created with new version number and the erroneous artifact may be deleted but not replaced. If the version number ends with -SNAPSHOT, then this uniqueness is not guaranteed or required. Snapshots are usually stored in separate repository and are not published for the world. Repositories contain the artifacts in directories that are organized in a defined way. When Maven runs, it can access different repositories using https protocol. Formerly, the http protocol was also used, and for non-paying customers, the central repository was available via http only. However, it was discovered that modules downloaded from the repository could be targets for men-in-the-middle security attacks and Sonatype (http://www.sonatype.com) changed the policy and used https protocol only. Never configure or use a repository with the http protocol. Never trust a file that you downloaded from HTTP. There is a local repository on the developer machine, usually under the ~/.m2/repository directory. When you issue the mvn install command, Maven stores the created artifact here. Maven also stores an artifact here when it is downloaded from a repository via HTTPS. This way, subsequent compilations do not need to go out to the network for the artifacts. Companies usually set up their own repository manager (the one that Sonatype, the company backing Maven, is providing Nexus). These applications can be configured to communicate with several other repositories and collect the artifacts from there on demand, essentially implementing proxy functionality. Artifacts travel to the build from the far end repositories to the closer ones in a hierarchical structure to the local repo and essentially to the final artifact if the packaging type of the project is war, ear, or some other format that encloses the dependent artifacts. This is essentially file caching without revalidation and cache eviction. This can be done because of the strict rules of artifact uniqueness. This is the reason for such a strict rule. If the project bubble were a standalone project, and not part of a multi-module one, then the dependency would look like this: <dependencies> <dependency> <groupId>packt.java9.by.example</groupId> <artifactId>SortInterface</artifactId> <version>1.0.0-SNAPSHOT</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> </dependency> </dependencies> If version is not defined for a dependency, Maven will not be able to identify which artifact to use. In the case of a multi-module project, version can be defined in the parent and the modules can inherit the version. As the parent is not dependent on the actual dependency, it only defines the version attached to the groupId and artifactId; the XML tag is not dependencies, but dependencyManagement/dependencies under the toplevel project tag as in the following example: <dependencyManagement> <dependencies> <dependency> <groupId>packt.java9.by.example</groupId> <artifactId>SortInterface</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> </dependencies> </dependencyManagement> If the parent POM uses the dependencies tag directly, Maven is not able to decide if the parent depends on that artifact or some modules. When the modules want to use junit, they need not specify the version. They will get it from the parent project defined as 4.12, which is the latest from JUnit 4. If ever there will be a new version 4.12.1, with some serious bugs fixed, then the only place to modify the version number is the parent POM, and the modules will use the new version starting with the next execution of the Maven compilation. When the new version, JUnit 5, comes out, however, the modules will all have to be modified because JUnit is not just a new version. Version 5 of JUnit is split into several modules and, this way, groupId and artifactId will also change. It is also worth noting that the modules that implement the interfaces from the SortInterface module are eventually dependent on this module. In this case, the version is defined as follows: <version>${project.version}</version> That seems to be a bit tautological (it is, actually). The ${project.version} property is the version of the project and it is inherited by the SortInterface module. This is the version of the artifact that the other modules depend on. In other words, the modules always depend on the version that we are currently developing. Code the sort To implement the sort, first, we will define the interfaces that a sort library should implement. Defining the interface before the actual coding is a good practice. When there are many implementations, it is sometimes recommended to first create a simple one and start using it so that the interface may evolve during the phase, and when the more complex implementations are due, then the interface to be implemented is already fixed, more or less. package packt.java9.by.example.ch03; <br/><br/>public interface Sort { <br/> void sort(SortableCollection collection); <br/>} package packt.java9.by.example.ch03; <br/><br/>public interface SortableCollection { <br/>} Creating BubbleSort Now, we can start creating the bubble sort that implements the Sort interface: package packt.java9.by.example.ch03.bubble; import packt.java9.by.example.ch03.*; import java.util.Comparator; public class BubbleSort implements Sort { @Override public void sort(SortableCollection collection) { int n = collection.size(); while (n > 1) { for (int j = 0; j < n - 1; j++) { if (comparator.compare(collection.get(j), collection.get(j + 1)) > 0) { swapper.swap(j, j + 1); } } n--; } } Normally, the algorithm to execute needs two operations that we implemented in the code last time specific to a String array: comparing two elements and swapping two elements. As this time the sort implementation itself does not know what type the elements are used and also does not know if the something it sorts is an array, a lists or something else, it needs something that does it for the sort when needed. More precisely, it needs a comparator object capable of comparing two elements and it needs a swapper object that is capable of swapping two elements in the collection. To get those, we can implement two setter methods that can set the objects for the purpose before sort is invoked. As this is not specific to the bubble sort algorithm but is rather general, these two methods should also be made a part of the interface, so the implementation is overriding it. private Comparator comparator = null; @Override public void setComparator(Comparator comparator) { this.comparator = comparator; } private Swapper swapper = null; @Override public void setSwapper(Swapper swapper) { this.swapper = swapper; } } The @Override annotation signals for the Java compiler that the method is overriding a method of the parent class, or, as in this case, of the interface. A method can override a parent method without this annotation; however, if we use the annotation, the compilation fails if the method does actually not override something. This helps you discover during compile time that something was changed in the parent class or in the interface and we did not follow that change in the implementation, or that we just made some mistake thinking that we will override a method when we actually do not. As annotations are heavily used in unit tests, we will talk about annotations in a bit more detail later. Amending the interfaces The modified Sort interface will look like this: public interface Sort { void sort(SortableCollection collection); void setSwapper(Swapper swap); void setComparator(Comparator compare); } This also means that we will need two new interfaces: Swapper and Comparator. We are lucky that the Java runtime already defines a Comparator interface that just fits the purpose. You may have guessed that from the following import statement: import java.util.Comparator; When you need something very basic, like a comparator interface, it is most probably defined in the runtime. It is advisable to consult the runtime before writing your own version. The Swapper interface, however, we will have to create. package packt.java9.by.example.ch03; public interface Swapper { void swap(int i, int j); } As it is used to swap two elements specified by the indices in SortableCollection, there is a method, quite trivially named swap for the purpose. But, we are not ready yet. If you try to compile the preceding code, the compiler will complain about the get and size methods. They are needed by the algorithm to implement the sort, but they are not inherently part of the sorting itself. This is a responsibility that should not be implemented in the sort. As we do not know what type of collections we will sort, it is not only unadvisable but also impossible to implement these functionalities inside the sort. It seems that we just cannot sort anything. There are some restrictions we will have to set. The sorting algorithm must know the size of the collection we sort and also should have access to an element by index so that it can pass it on to the comparator. These restrictions are expressed in the SortableCollection interface that we just left empty not knowing before the first sort implementation what is required to be there. package packt.java9.by.example.ch03; public interface SortableCollection { Object get(int i); int size(); } Now, we are ready with the interfaces and the implementation and we can go on testing the code. But, before that, we will briefly reiterate what we did and why we did that. Architectural considerations We created an interface and a simple implementation of it. During the implementation, we discovered that the interface needs other interfaces and methods that are needed to support the algorithm. This usually happens during the architectural design of the code, before implementation. For didactical reasons, I followed the build-up of the interfaces while we developed the code. In real life, when I created the interfaces, I created them all in one step as I have enough experience. I wrote my first quick sort code around 1983 in Fortran. However, it does not mean that I hit the bull's eye with just any problem and come out with the final solution. It just happens that sort is a too well known problem. If you need to modify the interfaces or other aspects of your design during development, do not feel embarrassed. It is a natural consequence and a proof that you understand things better and better as time goes by. If the architecture needs change, it is better to be done than not, and the sooner it is, the better. In real life enterprise environments, we will design interfaces just to learn during development that there were some aspects that we forgot. They are very true and bit more complex operations than sorting a collection. In the case of the sorting problem, we abstracted the something we want to sort to the most possible extreme. The Java build in sort can sort arrays or lists. If you want to sort something that is not a list or an array, you have to create a class that implements the java.util.List interface with more than 24 methods it requires to wrap your sortable object to make it sortable by the JDK sort. To be honest, that is not too many, and in a real-world project, I would consider that as an option. However, we do not, and cannot know, what methods of the interface the built-in sort uses. Those that are used should be functionally implemented and those that are not, can contain a simple return statement because they are just never invoked. A developer can consult the source code of the JDK and see what methods are actually used, but that is not the contract of the search implementation. It is not guaranteed that a new version will still use only those methods. If a new version starts to use a method that we implemented with a single return statement, the sort will magically fail. It is also an interesting performance question how the swapping of two elements is implemented by the search using only the List interface. There is no put(int, Object) method in the List interface. There is add(int Object), but that inserts a new element and it may be extremely costly (burning CPU, disk, energy) to push all elements of the list up if the objects are stored, for example, on disk. Furthermore, the next step may be removing the element after the one we just inserted, doing the costly process of moving the tail of the list again. That is, the trivial implementation of put(int,Object) that the sort may or may not follow. Again, this is something that should not be assumed. When developers use libraries, classes, and methods from the JDK, open source, or commercial libraries, the developers may consult the source code but they should not rely on the implementation. You should rely only on the contract and the definition of the API that the library comes with. When you implement an interface from some external library, and you do not need to implement some part of it, and create some dummy methods, feel the danger in the air. It is an ambush. It is likely that either the library is poor quality or you did not understand how to use it. In our case, we separated the swapping and the comparison from the sort. The collection should implement these operations and provide them for the sort. The contract is the interface, and to use the sort, you have to implement all methods of the interfaces we defined. The interface of Sort defines setters that set Swapper and Comparator. Having dependencies set that way may lead to a code that creates a new instance of a class implementing the Sort interface, but does not set Swapper and Comparator before invoking Sort. This will lead to NullPointerException the first time the Comparator is invoked (or when the Swapper is invoked in case the implementation invokes that first, which is not likely, but possible). The calling method should inject the dependencies before using the class. When it is done through setters, it is called setter injection. This terminology is heavily used when we use frameworks such as Spring, Guice, or some other container. Creating these service classes and injecting the instance into our classes is fairly similar all the time. Container implementations contain the functionality in a general way and provide configuration options to configure what instances are to be injected into what other objects. Usually, this leads to shorter, more flexible, and more readable code. However, dependency injection is not exclusive to containers. When we write the testing code in the next section, and invoke the setters, we actually do dependency injection. There is another way of dependency injection that avoids the problem of dependencies not being set. This is called constructor injection. The dependencies are final private fields with no values. Remember that these fields should get their final values by the time the constructor finishes. Constructor injection passes the injected values to the constructor as arguments and the constructor sets the fields. This way, the fields are guaranteed to be set by the time the object was constructed. This injection, however, cannot be defined in an interface. Now, we already have the code, and we know the considerations of how the interfaces were created. This is the time to do some testing. Creating unit tests When we write code, we should test it. No code has ever gone into production before at least doing some test runs. There are different levels of tests having different aims, technologies, industry practices, and names. Unit tests, as the name suggests, test a unit of code. Integration tests test how the units integrate together. Smoke tests test a limited set of the features just to see that the code is not totally broken. There are other tests, until the final test, which is the proof of the work: user acceptance test. Proof of the pudding is eating it. A code is good if the user accepts it. Many times, I tell juniors that the name user acceptance test is a bit misleading, because it is not the user who accepts the result of a project, but the customer. By definition, the customer is the person who pays the bill. Professional development is paid; otherwise, it is not professional. The terminology is, however, user acceptance test. It just happens that customers accept the project only if the users can use the program. When we develop in Java, unit test is testing standalone classes. In other words, in Java development, a unit is a class when we talk about unit tests. To furnish unit tests, we usually use the JUnit library. There are other libraries, such as TestNG, but JUnit is the most widely used, so we will use JUnit. To use it as a library, first, we will have to add it to the Maven POM as a dependency. Adding JUnit as dependency Recall that we have a multi-module project, and the dependency versions are maintained in the parent POM under the dependencyManagement tag. <dependencyManagement> <dependencies> ... <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> </dependencies> </dependencyManagement> The scope of the dependency is test, which means that this library is needed only to compile the test code and during the execution of the test. The JUnit library will not make its way to the final released product; there is no need for it. If you find the JUnit library in some deployed production Web Archive (WAR) file, suspect that somebody was not properly managing the scopes of the libraries. Maven supports the compilation and the execution of JUnit tests in the lifecycle of the project. If we want to execute the tests, only we can issue the mvn test command. The IDEs also support the execution of the unit tests. Usually, the same menu item that can be used to execute a class that has a public static main method can be used. If the class is a unit test utilizing JUnit, the IDE will recognize it and execute the tests and usually give a graphical feedback on what test was executing fine and which ones failed, and how. Writing the BubbleSortTest class The test classes are separated from the production classes. They go into the src/test/java directory. When we have a class named, for example, BubbleSort, then the test will be named BubbleSortTest. This convention helps the executing environment to separate the tests from those classes that do not contain tests but are needed to execute the tests. To test the sort implementation we have just created, we can furnish a class that contains, for now, a single canSortStrings method. Unit test method names are used to document the functionality being tested. As the JUnit framework invokes each and every method that has the @Test annotation, the name of the test is not referenced anywhere in our code. We can bravely use arbitrary long method names; it will not hinder readability at the place where the method is invoked. package packt.java9.by.example.ch03.bubble; // imports deleted from print public class BubbleSortTest { @Test public void canSortStrings() { ArrayList actualNames = new ArrayList(Arrays.asList( "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" )); The method contains ArrayList with the actual names that we have already gotten familiar with. As we have a sort implementation and interface that needs SortableCollection, we will create one backed up by ArrayList. SortableCollection namesCollection = new SortableCollection() { @Override public Object get(int i) { return actualNames.get(i); } @Override public int size() { return actualNames.size(); } }; We declared a new object that has the SortableCollection type, which is an interface. To instantiate something that implements SortableCollection, we will need a class. We cannot instantiate an interface. In this case, define the class in the place of the instantiation. This is called an anonymous class in Java. The name comes from the fact that the name of the new class is not defined in the source code. The Java compiler will automatically create a name for the new class, but that is not interesting for the programmers. We will simply write new SortableCollection() and provide the needed implementation immediately following between { and }. It is very convenient to define this anonymous class inside the method as, this way, it can access ArrayList without passing a reference to ArrayList in the class. As a matter of fact, the reference is needed, but the Java compiler automatically does this. The Java compiler, in this case, also takes care that automatic reference passing this way can only be done using variables that were initialized and will not change during the execution of the code after the instantiation of the anonymous class. The variable actualNames was set and it should not be changed in the method later. As a matter of fact, we can even define actualNames to be final and this would have been a requirement if we used Java 1.7 or earlier. Starting with 1.8, the requirement is that the variable is effectively final, but you need not declare it to be final. The next thing that we need is a Swapper implementation for ArrayList. In this case, we will define a whole class inside the method. It can also be an anonymous class, but this time I decided to use a named class to demonstrate that a class can be defined inside a method. Usually, we do not do that in production projects. class SwapActualNamesArrayElements implements Swapper { @Override public void swap(int i, int j) { final Object tmp = actualNames.get(i); actualNames.set(i,actualNames.get(j)); actualNames.set(j, tmp); } } Last, but not least, we will need a comparator before we can invoke the sort. As we have Strings to compare, this is easy and straightforward. Comparator stringCompare = new Comparator() { @Override public int compare(Object first, Object second) { final String f = (String) first; final String s = (String) second; return f.compareTo(s); } }; Having everything prepared for the sorting, we will finally need an instance of the Sort implementation, set the comparator and the swapper, and invoke the sort. Sort sort = new BubbleSort(); sort.setComparator(stringCompare); sort.setSwapper(new SwapActualNamesArrayElements()); sort.sort(namesCollection); The last, but most important part of the test is to assert that the result is the one that we expect. JUnit helps us do that with the aid of the Assert class. Assert.assertEquals(Arrays.asList("Abraham", "Dagobert", "Johnson", "Wilkinson", "Wilson"), actualNames); } } The call to assertEquals checks that the first argument, the expected result, equals the second argument, the sorted actualNames. If they differ, then AssertionError is thrown; otherwise, the test just finishes fine. Good unit tests Is this a good unit test? If you read it in a tutorial book like this, it has to be. Actually, it is not. It is a good code to demonstrate some of the tools that JUnit provides and some Java language features, but as a real JUnit test, I will not use it in a real life project. What makes a unit test good? To answer this question, we will have to find what the unit test is good for and what it is that we use it for. We will create unit tests to validate the operation of the units and to document. Unit tests are not to find bugs. Developers eventually use unit tests during debugging sessions but, many times, the testing code created for the debugging is a temporary one. When the bug is fixed, the code used to find it will not get into the source code. For every new bug, there should be a new test created that covers the functionality that was not properly working, but it is hardly the test code that is used to find the bug. This is because unit tests are mainly for documentation. You can document a class using JavaDoc, but the experience shows that the documentation often becomes outdated. The developers modify the code, but they do not modify the documentation, and the documentation becomes obsolete and misleading. Unit tests, however, are executed by the build system and if Continuous Integration (CI) is in use (and it should be, in a professional environment), then the build will be broken if a test fails, all developers will get mail notification about it, and it will drive the developer breaking the build to fix the code or the test. This way, the tests verify that continuous development did not break anything in the code or, at least, not something that can be discovered using unit tests. A good unit test is readable Our test is far from being readable. A test case is readable if you look at it and in 15 seconds you can tell what it does. It assumes, of course, some experience in Java on behalf of the reader, but you get the point. Our test is cluttered with support classes that are not core to the test. Our test also hardly validates that the code is working properly. It actually does not. There are some bugs in it that I put there deliberately, which we will locate and zap in the following sections. One single test that sorts a single String array is far from validating a sort implementation. If I were to extend this test to a real-world test, we would need methods that would have the name canSortEmptyCollection, canSortOneElementCollection, canSortTwoElements, canSortReverseOrder, or canSortAlreadySorted. If you look at the names, you will see what tests we need. Coming from the nature of the sort problem, an implementation may be reasonably sensitive to errors in these special cases. What are the good points in our unit test, in addition to it being an acceptable demonstration tool? Unit tests are fast Our unit test runs fast. As we execute unit tests each time, the CI fires up a build and the execution of the tests should not last long. You should not create a unit test sorting billions of elements. That is a kind of stability or load test and they should run in separate test periods and not every time the build is running. Our unit test sorts five elements that are reasonable. Unit tests are deterministic Our unit test is deterministic. Non-deterministic unit tests are the nightmare of the developers. If you are in a group where some builds break on the CI server, and when a build breaks, your fellow developer says that you just have to try it again; no way! If a unit test runs, it should run all times. If it fails, it should fail no matter how many times you start it. A non-deterministic unit test, in our case, will be to render random numbers and have them sorted. We will end up with different arrays in each test run and, in case there is some bug in the code that manifests for some array, we will not be able to reproduce it. Not to mention that the assertion that the code was running fine is also difficult. If we sorted a random array in a unit test (something we do not), we could, hypothetically, assert that the array is sorted, comparing the elements one after the other checking that they are in ascending order. It would also be a totally wrong practice. Assertions should be as simple as possible If the assertion is complex, the risk of introducing bugs in the assertion is higher. The more complex the assertion, the higher the risk. We will write the unit tests to ease our lives and not to have more code to debug. Additionally, one test should assert only one thing. This one assertion may be coded with multiple Assert class methods, one after the other. Still, the aim of these is to assert the correctness of one single feature of the unit. Remember the SRP: one test, one feature. A good test is like a good sniper: one shot, one kill. Unit tests are isolated When we test a unit A, any change in another unit B, or a bug in a different unit should not affect our unit test that is for the unit A. In our case, it was easy because we have only one unit. Later, when we develop the test for the quick sort, we will see that this separation is not that simple. If the unit tests are properly separated, a failing unit test clearly points out the location of the problem. It is in the unit where the unit test failed. If tests do not separate the units, then a failure in one test may be caused by a bug in a different unit than we expect. In this case, these tests are not really unit tests. In practice, you should make a balance. If the isolation of the units will be too costly, you can decide to create integration tests; and, if they still run fast, have them executed by the CI system. At the same time, you should also try to find out why the isolation is hard. If you cannot easily isolate the units in the tests, it means that the units are too strongly coupled, which may not be a good design. Unit tests cover the code Unit tests should test all usual and also all special cases of the functionality. If there is a special case of code that is not covered by the unit test, the code is in danger. In case of a sort implementation, the general case is sorting, say five elements. The special cases are much more numerous usually. How does our code behave if there is only one element or if there are no elements? What if there are two? What if the elements are in reverse order? What if they are already sorted? Usually, the special cases are not defined in the specification. The programmer has to think about it before coding, and some special cases are discovered during coding. The hard thing is that you just cannot tell if you covered all special cases and the functionality of the code. What you can tell is if all the lines of code were executed during the testing or not. If 90% of the code lines are executed during the tests, then the code coverage is 90%, which is fairly good in real life, but you should never be content with anything less than 100%. Code coverage is not the same as functional coverage, but there is a correlation. If the code coverage is less than 100%, then at least one of the following two statements is true: The functional coverage is not 100% There is unused code in the tested unit, which can just be deleted The code coverage can be measured, the functional coverage cannot. The tools and IDEs support code coverage measurement. These measurements are integrated into the editor so you will not only get the percentage of the coverage, but the editor will show you exactly which lines are not covered by the coverage coloring the lines (in Eclipse, for example) or the gutter on the left side of the editor window (IntelliJ). The picture shows that in IntelliJ, the tests cover the lines indicated by a green color on the gutter. (In the print version this is just a grey rectangle). Refactor the test Now that we have discussed what a good unit test is, let's improve our test. The first thing is to move the supporting classes to separate files. We will create ArrayListSortableCollection: package packt.java9.by.example.ch03.bubble; import packt.java9.by.example.ch03.SortableCollection; import java.util.ArrayList; public class ArrayListSortableCollection implements SortableCollection { final private ArrayList actualNames; ArrayListSortableCollection(ArrayList actualNames) { this.actualNames = actualNames; } @Override public Object get(int i) { return actualNames.get(i); } @Override public int size() { return actualNames.size(); } } This class encapsulates ArrayList and then implements the get and size methods to ArrayList access. ArrayList itself is declared as final. Recall that a final field has to be defined by the time the constructor finishes. This guarantees that the field is there when we start to use the object and that it does not change during the object lifetime. Note, however, that the content of the object, in this case, the elements of ArrayList, may change. If it were not the case, we would not be able to sort it. The next class is StringComparator. This is so simple that I will not list it here; I will leave it to you to implement the java.util.Comparator interface that can compare two Strings. It should not be difficult, especially as this class was already a part of the previous version of the BubbleSortTest class (hint: it was an anonymous class that we stored in the variable named stringCompare). We also have to implement ArrayListSwapper, which also should not be a big surprise. package packt.java9.by.example.ch03.bubble; import packt.java9.by.example.ch03.Swapper; import java.util.ArrayList; public class ArrayListSwapper implements Swapper { final private ArrayList actualNames; ArrayListSwapper(ArrayList actualNames) { this.actualNames = actualNames; } @Override public void swap(int i, int j) { Object tmp = actualNames.get(i); actualNames.set(i, actualNames.get(j)); actualNames.set(j, tmp); } } Finally, our test will look this: package packt.java9.by.example.ch03.bubble; // ... imports deleted from print ... public class BubbleSortTest { @Test public void canSortStrings() { ArrayList actualNames = new ArrayList(Arrays.asList( "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" )); ArrayList expectedResult = new ArrayList(Arrays.asList( "Abraham", "Dagobert", "Johnson", "Wilkinson", "Wilson" )); SortableCollection names = new ArrayListSortableCollection(actualNames); Sort sort = new BubbleSort(); sort.setComparator( new StringComparator()); sort.setSwapper( new ArrayListSwapper(actualNames)); sort.sort(names); Assert.assertEquals(expectedResult, actualNames); } } Now this is already a test that can be understood in 15 seconds. It documents well how to use a sort implementation that we defined. It still works and does not reveal any bug, as I promised. Collections with wrong elements The bug is not trivial, and as usual, this is not in the implementation of the algorithm, but rather in the definition, or the lack of it. What should the program do if there are not only strings in the collection that we sort? If I create a new test that starts with the following lines, it will throw ClassCastException: @Test public void canNotSortMixedElements() { ArrayList actualNames = new ArrayList(Arrays.asList( 42, "Wilson", "Wilkinson", "Abraham", "Dagobert" )); ... the rest of the code is the same as the previous test The problem here is that Java collections can contain any type of elements. You cannot ever be sure that a collection, such as ArrayList, contains only the types that you expect. Even if you use generics (we have not learned that, but we will in this chapter), the chances of a bug somehow conjuring up some object of an inappropriate type into a collection, are smaller but are still there. Don't ask me how; I cannot tell you. This is the nature of the bugs—you cannot tell how they work until you zap them. The thing is that you have to be prepared for such an exceptional case. Handling exceptions Exceptional cases should be handled in Java using exceptions. The ClassCastException is there and it happens when the sort tries to compare String to Integer using StringComparator, and to do that, it tries to cast an Integer to String. When an exception is thrown by the program using the throw command, or by the Java runtime, the execution of the program stops at that point, and instead of executing the next command, it continues where the exception is caught. It can be in the same method, or in some calling method up in the call chain. To catch an exception, the code throwing the exception should be inside a try block, and the catch statement following the try block should specify an exception that is compatible with the exception thrown. If the exception is not caught, then the Java runtime will print out the message of the exception along with a stack trace that will contain all the classes, methods, and line numbers on the call stack at the time of the exception. In our case, the mvn test command will produce the following trace in the output: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String at packt.java9.by.example.ch03.bubble.StringComparator.compare(StringComparator.java:9) at packt.java9.by.example.ch03.bubble.BubbleSort.sort(BubbleSort.java:13) at packt.java9.by.example.ch03.bubble.BubbleSortTest.canNotSortMixedElements(BubbleSortTest.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) ... some lines deleted from the print at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ... some lines deleted from the print at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) This stack trace is not really long. In the production environment in an application that runs on an application server, the stack trace may contain a few hundred elements. In this trace, you can see that Maven was starting the test execution, involved Maven surefire plugin, and then the JUnit executor, until we get through the test to the comparator, where the actual exception was thrown. This exception was not printed by the Java runtime to the console. This exception is caught by the JUnit library code and the stack trace is logged out to the console using Maven logging facility. The problem with this approach is that the real issue is not the class casting failure. The real issue is that the collection contains mixed elements. It is only realized by the Java runtime when it tries to cast two incompatible classes. Our code can be smarter. We can amend the comparator. package packt.java9.by.example.ch03.bubble; import java.util.Comparator; public class StringComparator implements Comparator { @Override public int compare(Object first, Object second) { try { final String f = (String) first; final String s = (String) second; return f.compareTo(s); } catch (ClassCastException cce) { throw new NonStringElementInCollectionException ( "There are mixed elements in the collection.", cce); } } } This code catches the ClassCastException and throws a new one. The advantage of throwing a new exception is that you can be sure that this exception is thrown from the comparator and that the problem really is that there are mixed elements in the collection. Class casting problems may happen at other places of the code as well, inside some of the sort implementations. Some application code may want to catch the exception and want to handle the case; for example, sending an application-specific error message and not dumping only a stack trace to the user. This code can catch ClassCastException as well, but it cannot be sure what the real cause of the exception is. On the other hand, NonStringElementInCollectionException is definite. The NonStringElementInCollectionException is an exception that does not exist in the JDK. We will have to create it. Exceptions are Java classes and our exception looks as follows: package packt.java9.by.example.ch03.bubble; public class NonStringElementInCollectionException extends RuntimeException { public NonStringElementInCollectionException (String message, Throwable cause) { super(message, cause); } } Java has the notion of checked exceptions. It means that any exception that is not extending RuntimeException should be declared in the method definition. Suppose our exception was declared as follows: public class NonStringElementInCollectionException extends Exception Then, we will have to declare the compare method as follows: public int compare(Object first, Object second) throws NonStringElementInCollectionException The problem is that the exception a method throws is part of the method signature, and this way compare will not override the compare method of the interface, and, that way, the class will not implement the Comparator interface. Thus, our exception has to be a runtime exception. There can be a hierarchy of exceptions in an application, and often, novice programmers create huge hierarchies of them. If there is something you can do, it does not mean that you should do it. Hierarchies should be kept as flat as possible, and this is especially true for exceptions. If there is an exception in the JDK that describes the exceptional case, then use the readymade exception. Just as well as for any other class: if it is ready, do not implement it again. It is also important to note that throwing an exception should only be done in exceptional cases. It is not to signal some normal operational condition. Doing that hinders readability of the code and also eats CPU. Throwing an exception is not an easy task for the JVM. It is not only the exception that can be thrown. The throw command can throw, and the catch command can catch anything that extends the Throwable class. There are two subclasses of Throwable: Error, and Exception. The Error exception is thrown if some error happened during the execution of the Java code. The two most infamous errors are OutOfMemoryError and StackOverflowError. If any of these happens, you cannot do anything reliably to catch the error. There is also InternalError and UnknownError in the JVM, but since JVM is fairly stable, you will hardly ever meet these errors. When any of those errors happen, try to debug the code and try to find out why you use that much memory or such deep method calls and try to optimize your solution. What I have just said about creating exception hierarchies is true again to catch errors. The fact that you can catch errors does not mean that you should. On the contrary, you should never catch an error and, especially, never ever catch a Throwable. This way, we handled this special case when some programmer accidentally writes 42 among the names, but will it be nicer if the error was identified during compile time? To do that, we will introduce generics. Just a last thought before we go there. What class behavior do we test with the canNotSortMixedElements unit test? The test is inside the BubbleSortTest test class, but the functionality is in the comparator implementation, StringComparator. This test checks something that is out of the scope of the unit test class. I can use it for demonstration purposes, but this is not a unit test. The real functionality of the sort implementation can be formulized this way: whatever exception the comparator throws is thrown by the sort implementation. You can try to write this unit test, or read on; we will have it in the next section. The StringComparator class does not have a test class because StringComparator is part of the test and we will never write a test for a test. Otherwise, we will sink into an endless rabbit hole. Generics The generics feature was introduced into Java in version 5. To start with an example, our Sortable interface until now was this: package packt.java9.by.example.ch03; public interface SortableCollection { Object get(int i); int size(); } After introducing generics, it will be as follows: package packt.java9.by.example.ch03; public interface SortableCollection<E> { E get(int i); int size(); } The E identifier denotes a type. It can be any type. It says that a class is a sortable collection if it implements the interface, namely the two methods— size and get. The get method should return something that is of type E, whatever E is. This may not make too much sense up until now, but you will soon get the point. After all, generics is a difficult topic. The Sort interface will become the following: package packt.java9.by.example.ch03; import java.util.Comparator; public interface Sort<E> { void sort(SortableCollection<E> collection); void setSwapper(Swapper swap); void setComparator(Comparator<E> compare); } This still does not provide much more value than the previous version without generics, but, at least, it does something. In the actual class implementing the Sort interface, Comparator should accept the same type that SortableCollection uses. It is not possible that SortableCollection works on strings and we inject a comparator for integers. The implementation of BubbleSort is as follows: package packt.java9.by.example.ch03.bubble; import packt.java9.by.example.ch03.*; import java.util.Comparator; public class BubbleSort<E> implements Sort<E> { @Override public void sort(SortableCollection<E> collection) { ... sort code same as before } private Comparator<E> comparator = null; @Override public void setComparator(Comparator<E> comparator) { this.comparator = comparator; } ... method swapper same as before } The real power of generics will come when we will write the tests. The first test does not change much, although with the generics, it is more definite. @Test public void canSortStrings() { ArrayList<String> actualNames = new ArrayList< >(Arrays.asList( "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" )); ArrayList<String> expectedResult = new ArrayList<>(Arrays.asList( "Abraham", "Dagobert", "Johnson", "Wilkinson", "Wilson" )); SortableCollection<String> names = new ArrayListSortableCollection<>(actualNames); Sort<String> sort = new BubbleSort<>(); sort.setComparator(String::compareTo); sort.setSwapper(new ArrayListSwapper<>(actualNames)); sort.sort(names); Assert.assertEquals(expectedResult, actualNames); } When we define ArrayList, we will also declare that the elements of the list will be strings. When we allocate the new ArrayList, there is no need to specify again that the elements are strings because it comes from the actual elements there. Each of them is a string; therefore, the compiler knows that the only thing that can come between the < and > character is String. The two characters < and >, without the type definition in between, is called diamond operator. The type is inferred. If you get used to generics, this code brings you more information on the types that the collections work on and the code becomes more readable. The readability and the extra information is not the only point. As we know that the Comparator argument is Comparator<String> now, we can use advanced features of Java available since Java 8 and can pass the String::compareTo method reference to the comparator setter. The second test is the important one for us now. This is the test which ensures that Sort does not interfere with the exception that the comparator throws. @Test(expected = RuntimeException.class) public void throwsWhateverComparatorDoes () { ArrayList<String> actualNames = new ArrayList<>(Arrays.asList( 42, "Wilson", "Wilkinson", "Abraham", "Dagobert" )); SortableCollection<String> names = new ArrayListSortableCollection<>(actualNames); Sort<String> sort = new BubbleSort<>(); sort.setComparator((String a, String b) -> { throw new RuntimeException(); }); final Swapper neverInvoked = null; sort.setSwapper(neverInvoked); sort.sort(names); } The thing is, that it does not even compile. The compiler says that it cannot infer the type of ArrayList<> on the third line. When all the arguments of the asList method were strings, the method returned a list of String elements and therefore the new operator was known to generate ArrayList<String>. This time, there is an integer, and thus, the compiler cannot infer that ArrayList<> is for String elements. To change the type definition from ArrayList<> to ArrayList<String> is not a cure. In that case, the compiler will complain about the value 42. This is the power of generics. When you use classes that have type parameters, the compiler can detect when you provide a value of the wrong type. To get the value into ArrayList to check that the implementation really throws an exception, we will have to conjure the value into it. We can try to replace the value 42 with an empty String and then add the following line which will still not compile: actualNames.set(0,42); The compiler will still know that the value you want to set in ArrayList is supposed to be String. To get the array with the Integer element, you will have to explicitly unlock the safety handle and pull the trigger, shooting yourself: ((ArrayList)actualNames).set(0,42); Now, the test looks like this: @Test(expected = RuntimeException.class) public void throwsWhateverComparatorDoes() { ArrayList<String> actualNames = new ArrayList<>(Arrays.asList( "", "Wilson", "Wilkinson", "Abraham", "Dagobert" )); ((ArrayList) actualNames).set(0, 42); SortableCollection<String> names = new ArrayListSortableCollection<>(actualNames); Sort<String> sort = new BubbleSort<>(); sort.setComparator((a, b) -> { throw new RuntimeException(); }); final Swapper neverInvoked = null; sort.setSwapper(neverInvoked); sort.sort(names); } We will set the Swapper to be null because it is never invoked. When I first wrote this code, it was evident to me. A few days later, I read the code and I stopped. Why is swapper null? Then I remembered in a second or two. But any time, when reading and understanding the code hicks up, I tend to think about refactoring. I can add a comment to the line saying //never invoked, but comments tend to remain there even when functionality changes. I learned it the hard way in 2006, when a wrong comment prevented me from seeing how the code was executing. I was reading the comment while debugging, instead of the code, and bug fixing took two days while the system was down. Instead of a comment, I tend to use constructs that make the code express what happens. The extra variable may make the class file a few bytes bigger, but it is optimized out by the JIT compiler so the final code does not run slower. The comparator that throws an exception was provided as a lambda expression. Lambda expressions can be used in cases where an anonymous class or named class will be used having only one simple method. Lambda expressions are anonymous methods stored in variables or passed in argument for later invocation. We will discuss the details of lambda expressions in Chapter 8, Extending our E-Commerce Application. For now, we will go on implementing QuickSort, and to do that, we will use the TDD methodology. Test Driven Development Test Driven Development (TDD) is a code writing approach when the developers first write a test based on the specification and then write the code. This is just the opposite that the developer community got used to. The conventional approach that we followed was to write the code and then write tests for it. To be honest, the real practice many times was to write the code and test it with ad-hoc tests and no unit tests at all. Being a professional, you will never do that, by the way. You always write tests. (And now, write it down a hundred times: I will always write tests.) One of the advantages of TDD is that the tests do not depend on the code. As the code does not exist at the creation of the test, developers cannot rely on the implementation of the unit and, thus, it cannot influence the test creation process. This is generally good. Unit tests should be black box tests as much as possible. Black box test is a test that does not take into account the implementation of the tested system. If a system is refactored, implemented in a different way, but the interface it provides toward the external world is the same, then the black box tests should run just fine. A white box test depends on the internal working of the system tested. When the code changes the white box test, the code may also need tuning to follow the change. The advantage of a white box test can be the simpler test code. Not always. Gray box test is a mixture of the two. Unit tests should be black box tests, but, many times, it is not simple to write a black box test. Developers will write a test that they think is black box, but many times, this belief proves to be false. When the implementation changes, something is refactored and the test does not work anymore and it needs to be corrected. It just happens that knowing the implementation, the developers, especially those who wrote the unit, will write a test that depends on the internal working of the code. Writing the test before the code is a tool to prevent this. If there is no code, you cannot depend on it. TDD also says that the development should be an iterative approach. You write only one test at the start. If you run, it fails. Of course it fails! As there is no code yet, it has to fail. Then, you will write the code that fulfills this test. Nothing more, only the code that makes this test pass. Then, you will go on writing a new test for another part of the specification. You will run it and it fails. This proves that the new test does test something that was not developed yet. Then, you will develop the code to satisfy the new test and, possibly, you will also modify a block of code that you have already written in the previous iterations. When the code is ready, the tests will pass. Many times, developers are reluctant to modify the code. This is because they are afraid of breaking something that was already working. When you follow TDD, you should not, and at the same time, you need not be afraid of this. There are tests for all features that were already developed. If some of the code modification breaks some functionality, the tests will immediately signal the error. The key is that you run the tests as often as possible when the code is modified. Implementing QuickSort Quick sort, as we have already discussed, is made of two major parts. One is partitioning and the other one is doing the partitioning recursively until the whole array is sorted. To make our code modular and ready to demonstrate the Java 9 module-handling feature, we will develop the partitioning and the recursive sorting into separate classes and in a separate package. The complexity of the code will not justify this separation. The partitioning class The partitioning class should provide a method that moves the elements of the collection based on a pivot element, and we will need to know the position of the pivot element after the method finishes. The signature of the method should look something like this: public int partition(SortableCollection<E> sortable, int start, int end, E pivot); The class should also have access to Swapper and Comparator. In this case, we defined a class and not an interface; therefore, we will use constructor injection. These constructs, like setters and constructor injectors, are so common and happen so frequently that IDEs support the generation of these. You will need to create the final fields in the code and use the code generation menu to create the constructor. The partitioning class will look like the following: package packt.java9.by.example.ch03.qsort; import packt.java9.by.example.ch03.SortableCollection; import packt.java9.by.example.ch03.Swapper; import java.util.Comparator; public class Partitioner<E> { private final Comparator<E> comparator; private final Swapper swapper; public Partitioner(Comparator<E> comparator, Swapper swapper){ this.comparator = comparator; this.swapper = swapper; } public int partition(SortableCollection<E> sortable, int start, int end, E pivot){ return 0; } } This code does nothing, but that is how TDD starts. We will create the definition of a requirement providing the skeleton of the code and the test that will call it. To do that, we will need something that we can partition. The simplest choice is an Integer array. The partition method needs a object of type SortableCollection<E>, and we will need something that wraps the array and implements this interface. We name that class ArrayWrapper. This class serves a general purpose and it is not only for the test. Because of that, we create it as production code and as such we put it in the directory main and not in the directory test. As this wrapper is independent from the implementation of Sort, the proper position of this class is in a new SortSupportClasses module. We will create the new module as it is not part of the interface. Implementations depend on the interface, but not on the support classes. There can also be some application that uses our libraries and may need the interface module and some of the implementation but still does not need the support classes when they deliver the wrapping functionality themselves. After all, we cannot implement all possible wrapping functionality. The SRP also holds for the modules. Java libraries tend to contain unrelated functionalities. For the short run, it makes the use of the library simpler. You will only need to specify one dependency in your POM file and you will have all the classes and APIs that you need. In the long run, the application gets bigger, carrying a lot of classes that are part of some of the libraries but the application never uses them. To add the new module, the module directory has to be created along with the source directories and the POM file. The module has to be added to the parent POM and it also has to be added to the dependencyManagement section so that the test code of the QuickSort module can use it without specifying the version. The new module depends on the interface module, so this dependency has to be added to the POM of the support classes. The ArrayWrapper class is simple and general. package packt.java9.by.example.ch03.support; import packt.java9.by.example.ch03.SortableCollection; public class ArrayWrapper<E> implements SortableCollection<E> { private final E[] array; public ArrayWrapper(E[] array) { this.array = array; } public E[] getArray() { return array; } @Override public E get(int i) { return array[i]; } @Override public int size() { return array.length; } } The ArraySwapper class, which we also need, comes into the same module. It is just as simple as the wrapper. package packt.java9.by.example.ch03.support; import packt.java9.by.example.ch03.Swapper; public class ArraySwapper<E> implements Swapper { private final E[] array; public ArraySwapper(E[] array) { this.array = array; } @Override public void swap(int k, int r) { final E tmp = array[k]; array[k] = array[r]; array[r] = tmp; } } Having these classes, we can create our first test. package packt.java9.by.example.ch03.qsort; // imports deleted from print public class PartitionerTest { Before creating the @Test method, we will need two helper methods that make assertions. Assertions are not always simple, and in some cases, they may involve some coding. The general rule is that the test and the assertions in it should be as simple as possible; otherwise, they are just possible source of programming errors. Additionally, we created them to avoid programming errors, not to create new ones. The assertSmallElements method asserts that all elements before cutIndex are smaller than pivot. private void assertSmallElements(Integer[] array, int cutIndex, Integer pivot) { for (int i = 0; i < cutIndex; i++) { Assert.assertTrue(array[i] < pivot); } } The assertLargeElements method makes sure that all elements following cutIndex are at least as large as pivot. private void assertLargeElemenents(Integer[] array, int cutIndex, Integer pivot) { for (int i = cutIndex; i < array.length; i++) { Assert.assertTrue(pivot <= array[i]); } } The test uses a constant array of Integers and wraps it into an ArrayWrapper class. @Test public void partitionsIntArray() { Integer[] partitionThis = new Integer[]{0, 7, 6}; Swapper swapper = new ArraySwapper<>(partitionThis); Partitioner<Integer> partitioner = new Partitioner<>((a, b) -> a < b ? -1 : a > b ? +1 : 0, swapper); final Integer pivot = 6; final int cutIndex = partitioner.partition(new ArrayWrapper<>(partitionThis), 0, 2, pivot); Assert.assertEquals(1, cutIndex); assertSmallElements(partitionThis, cutIndex, pivot); assertLargeElemenents(partitionThis, cutIndex, pivot); } } There is no Comparator for Integer type in the JDK, but it is easy to define one as a lambda function. Now we can write the partition method, as follows: public int partition(SortableCollection<E> sortable, int start, int end, E pivot){ int small = start; int large = end; while( large > small ){ while( comparator.compare(sortable.get(small), pivot) < 0 && small < large ){ small ++; } while( comparator.compare(sortable.get(large), pivot) >= 0 && small < large ){ large--; } if( small < large ){ swapper.swap(small, large); } } return large; } If we run the test, it runs fine. However, if we run the test with coverage, then the IDE tells us that the coverage is only 92%. The test covered only 13 of the 14 lines of the partition method. There is a red rectangle on the gutter at line 28. This is because the test array is already partitioned. There is no need to swap any element in it when the pivot value is 6. It means that our test is good, but not good enough. What if there is an error on that line? To amend this problem, we will extend the test, changing the test array from { 0, 7, 6 } to { 0, 7, 6, 2}. Run the test and it fails. Why? After some debugging, we will realize that we invoke the method partition with the fixed parameter 2 as the last index of the array. But, we made the array longer. Why did we write a constant there in the first place? It is a bad practice. Let's replace it with partitionThis.length-1. Now, it says that cutIndex is 2, but we expected 1. We forgot to adjust the assertion to the new array. Let's fix it. Now it works. The last thing is to rethink the assertions. The less code the better. The assertion methods are quite general, and we will use it for one single test array. The assertion methods are so complex that they deserve their own test. But, we do not write code to test. Instead of that, we can simply delete the methods and have the final version of the test. @Test public void partitionsIntArray() { Integer[] partitionThis = new Integer[]{0, 7, 6, 2}; Swapper swapper = new ArraySwapper<>(partitionThis); Partitioner<Integer> partitioner = new Partitioner<>((a, b) -> a < b ? -1 : a > b ? +1 : 0, swapper); final Integer pivot = 6; final int cutIndex = partitioner.partition(new ArrayWrapper<>(partitionThis), 0, partitionThis.length-1, pivot); Assert.assertEquals(2, cutIndex); final Integer[] expected = new Integer[]{0, 2, 6, 7}; Assert.assertArrayEquals(expected,partitionThis); } And then again, is this a black-box test? What if the partitioning returns {2, 1, 7, 6}? It fits the definition. We can create more complex tests to cover such cases. But a more complex test may also have a bug in the test itself. As a different approach, we can create tests that may be simpler but rely on the internal structure of the implementation. These are not black-box tests and thus not ideal unit tests. I will go for the second one, but I will not argue if someone chooses the other. Recursive sorting We will implement the quick sort with an extra class that is in the qsort package along with the partitioning class, which is as follows: package packt.java9.by.example.ch03.qsort; // imports deleted from the print public class Qsort<E> { // constructor injected final fields deleted from the print public void qsort(SortableCollection<E> sortable, int start, int end) { if (start < end) { final E pivot = sortable.get(start); final Partitioner<E> partitioner = new Partitioner<>(comparator, swapper); int cutIndex = partitioner.partition(sortable, start, end, pivot); if (cutIndex == start) { cutIndex++; } qsort(sortable, start, cutIndex - 1); qsort(sortable, cutIndex, end); } } } The method gets SortableCollection<E> and two index parameters. It does not sort the whole collection; it sorts only the elements between the start and the end index. It is always important to be extremely precise with the indexing. Usually, there is no problem with the start index in Java, but a lot of bugs source from how the end index is interpreted. In this method, the value of end can mean that the index is already not part of the to-besorted interval. In that case, the partition method should be invoked with end-1 and the first recursive call with cutIndex as last parameter. It is a matter of taste. The important thing is to be precise and define the interpretation of index parameters. If there is only one element (start == end), then there is nothing to be sorted and the method returns. This is the end criterion of the recursion. The method also assumes that the end index is never smaller than the start index. As this method is used only inside the library that we are developing at the moment, such an assumption is not too risky to make. If there is something to be sorted, then the method takes the first element of the to-be-sorted interval and uses it as pivot and calls the partition method. When the partition is done, the method recursively calls itself for the two halves. This algorithm is recursive. This means that the method calls itself. When a method call is executed, the processor allocates some memory in an area called stack and it stores the local variables there. This area that belongs to the method in the stack is called stack frame. When the method returns, this area is released and the stack is restored, simply moving the stack pointer where it was to the previous state. This way a method can continue its execution after calling another method; the local variables are there. When a method calls itself, it is not different. The local variables are local to the actual call of the method. When the method calls itself, it allocates space for the local variables again on the stack. In other words, these are new instances of the local variables. We will use recursive methods in Java, and in other programming languages, when the definition of the algorithm is recursive. It is extremely important to understand that when the processor code runs, it is not recursive any more. On that level, there are instructions, register stores, and memory loads and jumps. There is nothing like function or method and therefore, on that level, there is nothing like recursion. If you get that, it is easy to understand that any recursion can be coded as a loop. As a matter of fact, it is also true the other way around—every loop can be coded as recursion but that is not really interesting until you start functional programming. The problem with the recursion in Java, and in many other programming languages, is that it may run out of stack space. In the case of quick sort, this is not the case. You can safely assume that the stack for method calling in Java is a few hundreds of levels. Quick sort needs a stack that is approximately log2n deep, where n is the number of elements to be sorted. In the case of one billion elements, this is 30 that should just fit. Why is the stack not moved or resized? That is because the code that runs out of the stack space is usually bad style. They can be expressed more readable in form of some loop. A more robust stack implementation would only lure the novice programmer to do some less readable recursive coding. There is a special case of recursion named tail recursion. A tail recursive method calls itself as the last instruction of the method. When the recursive call returns the code, executing the method does nothing else but release the stack frame that was used for this method invocation. In other words, we will keep the stack frame during the recursive call just to throw it away afterwards. Why not throw it away before the call? In that case, the actual frame, which has the same size and call, will allocate because this is just the same method that is kept and the recursive call is transformed into a jump instruction. This is an optimization that Java does not do. Functional languages are doing it, but Java is not really a functional language and therefore tail-recursive functions should rather be avoided and transformed to a loop in the Java source level. Non-recursive sorting To demonstrate that even non-tail recursive methods can be expressed in a non-recursive way, here is the quick sort that way: public class NonRecursiveQuickSort<E> { // injected final fields and constructor deleted from print private static class Stack { final int begin; final int fin; public Stack(int begin, int fin) { this.begin = begin; this.fin = fin; } } public void qsort(SortableCollection<E> sortable, int start, int end) { final List<Stack> stack = new LinkedList<>(); final Partitioner<E> partitioner = new Partitioner<>(comparator, swapper); stack.add(new Stack(start, end)); int i = 1; while (!stack.isEmpty()) { Stack iter = stack.remove(0); if (iter.begin < iter.fin) { final E pivot = sortable.get(iter.begin); int cutIndex = partitioner.partition(sortable, iter.begin, iter.fin, pivot); if( cutIndex == iter.begin ){ cutIndex++; } stack.add(new Stack(iter.begin, cutIndex - 1)); stack.add(new Stack(cutIndex, iter.fin)); } } } } This code implements a stack on the Java level. While it sees that there is still something scheduled to be sorted in stack, it fetched it from the stack and does the sort partitioning, and schedules the two parts for being sorted. This code is more complex than the previous one and you have to understand the role of the Stack class and how it works. On the other hand, the program uses only one instance of the Partitioner class and it is also possible to use a thread pool to schedule the subsequent sorts instead of handling the tasks in a single process. This may speed up the sort when it is executed on a multi-CPU machine. However, this is a bit more complex task and this chapter contains a lot of new things without multitasking; therefore, we will look at multithread code in two chapters later only. In the very first version of the sort, I was coding it without the three lines that compare cutIndex against the interval start and increments it in the if branch. It is needed very much. But, the unit tests we created in this book do not discover the bug if we miss those lines. I recommend that you just delete those lines and try to write some unit tests that fail. Then try to understand what the special case is when those lines are vital and try to modify your unit test so that it is the simplest possible that still discovers that bug. (Finally, put the four lines back and see if the code works.) Additionally, find some architectural reason why not to put this modification into the method partition. That method could just return large+1 in case large == start. Implementing the API class Having done all this, the last thing we will need is to have QuickSort as a simple class (all the real work was already done in different classes). public class QuickSort<E> implements Sort<E> { public void sort(SortableCollection<E> sortable) { int n = sortable.size(); Qsort<E> qsort = new Qsort<>(comparator,swapper); qsort.qsort(sortable, 0, n-1); } // ... setter injectors were deleted from the print } Do not forget that we also need a test! But, in this case, that is not much different than that of BubbleSort. @Test public void canSortStrings() { final String[] actualNames = new String[]{ "Johnson", "Wilson", "Wilkinson", "Abraham", "Dagobert" }; final String[] expected = new String[]{"Abraham", "Dagobert", "Johnson", "Wilkinson", "Wilson"}; Sort<String> sort = new QuickSort<>(); sort.setComparator(String::compareTo); sort.setSwapper(new ArraySwapper<String>(actualNames)); sort.sort(new ArrayWrapper<>(actualNames)); Assert.assertArrayEquals(expected, actualNames); } This time, we used String array instead of ArrayList. This makes this test simpler and, this time, we already have the support classes. You may recognize that this is not a unit test. In the case of BubbleSort, the algorithm was implemented in a single class. Testing that single class is a unit test. In the case of QuickSort, we separated the functionality into separate classes, and even into separate packages. A real unit test of the QuickSort class will disclose the dependency of that class on other classes. When this test runs, it involves the execution of Partitioner and also Qsort; therefore, it is not really a unit test. Should we bother about that? Not really. We want to create unit tests that involve a single unit to know where the problem is when a unit test fails. If there were only integration tests, a failing test case would not help a lot in pointing out where the problem is. All it says is that there is some problem in the classes that are involved in the test. In this case, there are only a limited number of classes (three) that are involved in this test and they are tied together. They are actually tied together and related to each other so closely that in the real production code, I would have implemented them in a single class. I separated them here to demonstrate how to test a single unit and also to demonstrate Java 9 module support that needs a bit more than a single class in a JAR file. Creating modules Module handling, also known as project Jigsaw, is a feature that was made available only in Java 9. It was a long planned feature that the developers were waiting for. First it was planned for Java 7, but it was so complex that it got postponed to Java 8 and then to Java 9. A year ago, it seemed that it would get postponed again, but finally, the project code got into the early releases and now nothing can stop from being part of the release. Why modules are needed We have already seen that there are four levels of access in Java. A method or field can be private, protected, public, or default (also known as package private) when no modifier is supplied. When you develop a complex library to be used in several projects, the library itself will contain many classes in many packages. There will certainly be classes and methods, fields in those that are used inside the library by other classes from different packages, but classes that are not to be used by the code outside the library. Making them anything less visible than public will render them unusable inside the library. Making them public will make them visible from outside. In our code, the Maven module quick compiled to a JAR can only be used if the method sort can invoke qsort. But, we do not want qsort to be used directly from outside. In the next version, we may want to develop a version of the sort that uses qsort from the NonRecursiveQuickSort class and we do not want complaining customers whose code does not compile or work because of a minor library upgrade. We can document that the internal methods and classes are still public but not for use, but in vain. Developers using our library do not read documentation. This is also why we do not write excessive comments. Nobody will read it, not even the processor executing the code. The most well-known and infamous example of this problem is the sun.misc.Unsafe class in the JDK. There is some really unsafe code in it, as the name implies. You can access memory out of heap, create objects without initialization, and so on. You should not. Why bother? You are a well-behaving developer and you just stick to the rules and you do not use that package. Whenever it changes in a new version of the JDK, your program is safe using only public and well-documented JDK API. Right? Wrong! Without being aware of this, you may use some libraries that depend on other libraries that use the package. Mockito and Spring Framework are only two of the numerous in danger. In addition, Java 9 will definitely come with a new version of this package. However, it will also come with module handling. While Java 9 will provide some useful API for the libraries that were using the Unsafe package because there was no provided API for the functionality they needed, it will deliver modules not to recreate the same problem again. What is a Java module A Java module is a collection of classes in a JAR or in a directory that also contain a special class named module-info. If there is this file in a JAR or directory then it is a module, otherwise it is just a collection of classes that are on the classpath (or not). Java 8, and the earlier versions, will just ignore that class as it is never used as code. This way, using older Java, causes no harm and backward compatibility is maintained. The module information defines what the module exports and what it requires. It has a special format. For example, we can place module-info.java in our SortInterface Maven module. module packt.java9.by.example.ch03{ exports packt.java9.by.example.ch03; } This means that any class, which is public and inside the packt.java9.by.example.ch03 package, can be used from outside. This package is exported from the module, but other classes from other packages are not visible from outside of the module even if they are public. The name of the module is same as the package, but this is mere convention in case there is only one package exported. The requirement is the same as in the case of packages: there should be a name that is not likely to collide with other module names. The reversed domain name is a good choice but it is not a must as you can see in this book. There is no toplevel domain packt, yet. We should also configure the parent POM to ensure that the compiler we use is Java 9, <build> ... <plugins> ... <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.5.1</version> <configuration> <source>1.9</source> <target>1.9</target> </configuration> </plugin> ... Older versions would be confused with the module-info.java file. (By the way, even the early access version of Java 9 I use for this book sometimes gives a hard time.) We also create a module-info.java file in the Maven module, quick, which is as follows: module packt.java9.by.example.ch03.quick{ exports packt.java9.by.example.ch03.quick; requires packt.java9.by.example.ch03; } This module exports another package and requires the packt.java9.by.example.ch03 module that we have just created. Now, we can compile the modules and the created JARs in the./quick/target and ./SortInterface/target directories are now Java 9 modules. As Maven does not fully support the modules yet, when I issue the mvn install command, I get the following error message: [ERROR] .../genericsort/quick/src/main/java/module-info.java:[3,40] module not found: packt.java9.by.example.ch03 Maven puts the compiled modules on classpath, but Java 9 seeks modulepath for modules. Maven does not handle modulepath yet. To hack modulepath to the compiler, we will have to add the following configuration lines to the parent POM to the configuration of the compiler plugin:<compilerArgs> <arg>-modulepath</arg> <arg>${project.parent.basedir}/SortInterface/target/SortInterface-1.0.0-SNAPSHOT.jar: ...</arg> </compilerArgs> The actual file should list all the colon separated JAR files that Maven generates, and on which some of the modules depend. These are the SortInterface, quick, and SortSupportClasses. To test the functionality of module support, we will create another Maven module called Main. It has only one class, called Main, with a public static void main method: package packt.java9.by.example.ch03.main; // ... imports deleted from the print public class Main { public static void main(String[] args) throws IOException { String fileName = args[0]; BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(new File(fileName)))); List<String> lines = new LinkedList<>(); String line; while ((line = br.readLine()) != null) { lines.add(line); } br.close(); String[] lineArray = lines.toArray(new String[0]); Sort<String> sort = new QuickSort<>(); Qsort<String> qsort = new Qsort<>(String::compareTo,new ArraySwapper<>(lineArray)); sort.setComparator(String::compareTo); sort.setSwapper(new ArraySwapper<>(lineArray)); sort.sort(new ArrayWrapper<>(lineArray)); for (final String outLine : lineArray) { System.out.println(outLine); } } } It takes the first argument (without checking that there is one, which we should not use in a production code) and uses that as a file name. Then, it reads the lines of the file into a String array, sorts it, and prints it to the standard output. As the module support only works for modules, this Maven module also has to be a Java module and have a module-info.java file. module packt.java9.by.example.ch03.main{ requires packt.java9.by.example.ch03.quick; requires packt.java9.by.example.ch03; requires packt.java9.by.example.ch03.support; } Additionally, we will have to create a module-info.java file for the support module; otherwise, we will not be able to use it from our module. After compiling the modules using mvn , we can run it to print out the parent POM. install java -cp Main/target/Main-1.0.0-SNAPSHOT.jar:SortInterface/target/SortInterface-1.0.0-SNAPSHOT.jar:quick/target/quick Note that this is one line of command that print breaks into several lines. Now, if we try to access Qsort directly inserting the following line Qsort<String> qsort = new Qsort<> (String::compareTo,new ArraySwapper<>(lineArray)); into the main method, Maven will complain because the module system hides it from our Main class: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on proj [ERROR] .../Main/src/main/java/packt/java9/by/example/ch03/main/Main.java:[4,41] package packt.java9.by.example.ch03. [ERROR] .../Main/src/main/java/packt/java9/by/example/ch03/main/Main.java:[25,9] cannot find symbol The module system also supports the java.util.ServiceLoader based class-loading mechanism, which we will not discuss in this book. This is an old technology that is rarely used in an enterprise environment when Spring, Guice, or some other dependency injection framework is used. If you see a module-info.java file that contains the uses and provides keywords, then first consult with the Java documentation about the ServiceLoader class at http://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html, and then the Java 9 language documentation on module support (http://openjdk.java.net/projects/jigsaw/quick-start). Summary In this chapter, we developed a general sorting algorithm implementing quick sort. We modified our project to be a multi-module Maven project and also to use Java module definitions. We were using JUnit to develop unit tests, and we developed the code using TDD. We converted the code from old style Java to new using generics, and we used exception handling. These are the basic tools that are needed for the coming chapters, where we will develop a guessing game. First we will develop a simpler version and in the subsequent chapter we will develop a version that uses parallel computing, and multiple proccessors. Mastermind - Creating a Game In this chapter, we will start to develop a simple game. The game is the Mastermind game for two players. Player one selects four differently colored pins out of six possible colors and arranges them on a board in a row hidden from the other player. The other player tries to guess the colors of the pins and its positions. After each try, player one tells the number of matching colors and the pins matching both color and position. The program will act as both player one and player two. Our code will play alone. However, what remains for us to play with is the most important: the code. This example is complex enough to deepen the OO principles and how we design classes and model the real world. We have already used classes provided in the Java runtime. This time, we will use collections and discuss this important area. These classes and interfaces are widely used and available in the JDK and as important for a professional Java developer as the language itself. The build tool this time is Gradle. In this chapter we will cover: Java collections Dependency injection How to comment our code and to create JavaDoc documentation How to create integration tests The Game Mastermind (https://en.wikipedia.org/wiki/Mastermind_(board_game)) is an old game. The plastic version that was ubiquitous in every house with children was invented in 1970. I got a board around 1980 as a Christmas gift and solving the game puzzle in BASIC language was one of the first programs that I created around 1984. The game board contains holes in several rows in four columns. There are plastic pins of six different colors that can be inserted into the holes. Each pin has one color. They are usually red, green, blue, yellow, black, and white. There is a special row that is hidden from one of the players (the guesser). To play the game, one of the players (hider) has to select four pins from a set of pins. The selected pins should have different colors. The pins are placed in the hidden row one by one, each into a position. The guesser tries to find out what colors are in which position guessing. Each guess takes place selecting four pins and placing them in a row. The hider tells the guesser how many pins are in correct position and how many have a color that is on the table, but are not in the position where that color is hidden. A sample play may go like this: The hider hides four pins with color blue, yellow, white, and black. Guesser guesses yellow, blue, green, and red. The hider tells the guesser that there are two colors matching, but none of them is in the position in the hidden row. The hider says this because yellow and blue are in the hidden row but not in the positions as the guesser guessed. They are actually swapped, but this information the hider keeps a secret. All she says is that there are two colors matching, none in the correct position. The next guess is ... The game finishes when the guesser finds the correct colors in the correct order. The same game, as on the figure, can also be described with textual notation, B for blue, Y for yellow, G for green, W for white, R for red, and b for black (lucky we have upper and lower case letters on the computer). RGBY 0/0 GRWb 0/2 YBbW BYGR RGYB RGBY 0/2 0/4 2/2 4/0 Guess what! This is the actual output of the program that we develop in this chapter. The model of the game When we develop a piece of code with an object-oriented mindset, we try to model the real world and map real-world objects to objects in the program. You certainly have heard of object orientation explained with the very typical examples of geometric objects, or the car and the motor thing to explain composition. Personally, I believe that these examples are too simple to get a good understanding. They may be good for starters, but we are already in the fourth chapter of the book. The Mastermind game is much better. It is a bit more complex than just rectangles and triangles, but not as complex as a telecom billing application or an atomic power plant control. What are the real-world objects that we have in that game? We have a table and we have pins of different colors. There are two Java classes that we certainly will need. What is in a table? There are rows each having four positions. Perhaps we will need a class for a row. A table will have rows. We will also need something that hides the secret. This also may be a row and each row may also hold the information about how many positions and how many colors are matching. In case of the secret row, this information is obvious: 4 and 0. What is a pin? Each pin has a color and generally, that is it. There are no other features of a pin, except that it can be inserted into a hole on the table, but this is a real life feature we will not model. Essentially, a pin is a color and nothing else. This way, we can eliminate the pin class from our model early on, even before we created it in Java. Instead, we have colors. What is a color? This is something that may be hard to immerse into the first time. We all know well what a color is. It is a mixture of different frequency of lights, as our eyes perceive it. We can have paints and prints in different colors, and so on. There are very many things that we do not model in this program. It is really hard to tell what we model about color in our code because these features are so obvious that we take it for granted in real life; we can tell about two colors that they are different. This is the only feature we need. To do this, the simplest class of Java can be used: package packt.java9.by.example.mastermind; public class Color {} If you have two variables of the type Color, you can tell if they are the same or not. You can use object identity comparing a and b using the expression a == b or you can use the equals method inherited from the Object class, a.equals(b). It is tempting to encode the colors with letters, or use String constants to denote them. It may be easier first, but there are serious drawbacks later. When the code becomes complex, it leads to bugs; it will be easy to pass something also encoded as String instead of a color and only unit tests may save the day. Better, the compiler already complains in the IDE when you type the wrong argument. When we play the game, the pins are in small boxes. We pull pins out of the boxes. How do we get the colors in the program? We need something from where we can fetch colors or looking at the other way something that can give us colors. We will call it ColorManager. ColorManager knows how many different colors we have and any time we need a color, we can ask for it. Again, there is a temptation to design the ColorManager that it can serve a color by its serial number. If we have four colors, we could ask for color number 0, 1, 2, or 3. But then again, it would just implicitly encode the colors as integer numbers, which we agreed we will not. We should find the minimum feature that we will need to model the game. To describe the structure of the classes, professional developers usually use UML class diagrams. UML is a diagram notation that is standardized and is almost exclusively used to visualize software architecture. There are many diagram types in UML to describe the static structure and the dynamic behavior of a program. This time, we will look at a very simplified class diagram. We have no room to get into the details of UML class diagrams. Rectangles denote the classes, normal arrows denote the relations when a class has field of the other class type, and triangle headed arrow means that a class extends another. The arrow points to the direction of the class being extended. A Game contains a secret Row and a Table. The Table has a ColorManager and a List<> of Row. The ColorManager has a first color and has a Map<> of Color. We have not discussed why that is the design, we will get there and the diagram helps us walking that road. A Row is essentially an array of Color. The one who plays the game has one function: it has to guess many times until it finds the hidden secret. To get to the model of the ColorManager, we will have to design the algorithm of the Guesser. When the player makes the first guess, any combination of colors is just as good as any other. Later, the guesses should consider the responses that were given for previous guesses. It is a reasonable approach to try only color variations that can be the actual secret. The player selects a variation and looks at all previous guesses assuming that the selected variation is the secret. If the responses to the rows he has already made are the same for this variation as for the unknown secret in the game, then it is reasonable to try this variation. If there is any difference in the responses, then this variation is certainly not the variation that was hidden. To follow this approach, the guesser has to generate all possible color variations one after the other and compare it against the table. The guesser code will not create and store all the possible variations ahead, but it has to know where it was and has to be able to calculate the next variation that comes. This assumes an order of the variations. For a short while, let's forget that no color may appear twice in a variation. A simple ordering can be made the same way as we sort decimal numbers. If we have a three-digit number, then the first one is 000, the next one is 001, and so on until 009, always fetching the next digit for the last position. After that, 010 comes. We increased a digit next to the last one and we set the last one to 0 again. Now, we have 011, 012, and so on. You know, how we count numbers. Now, replace the digits with colors and we have only six and not ten. Or, we have as many as we want when we instantiate a ColorManager object. This leads to the functionality of the ColorManager. It has to do the following two things: Give the first color to the caller Give the next color that follows a given color (we will name the method nextColor) The latter functionality should also signal some way when there is no next color. This will be implemented using another method, named thereIsNextColor. It is a convention to start the method names that return a Boolean value with is. That would lead to the name following this convention isThereNextColor, or isNextColor. Either of these names explains the functionality of the method. If I ask the question isThereNextColor, the method will answer me true or false. But, this is not how we will use the method. We will talk in simple sentences. We will use short sentences. We will avoid unnecessary, gibberish expressions. We will also program that way. Most probably, the caller will use this method in an if statement. They will write the following: If( thereIsNextColor(currentColor)){...} and not if( isThereNextColor(currentColor)){...} I think the first version is more readable and readability comes first. Last, but not least, nobody will blame you if you follow the old convention, and in case that is the company standard, you have to anyway. To do these, the ColorManager also has to create the color objects and should store them in a structure that helps the operations being performed. package packt.java9.by.example.mastermind; import java.util.HashMap; import java.util.Map; public class ColorManager { final protected int nrColors; final protected Map<Color, Color> successor = new HashMap<>(); final private Color first; public ColorManager(int nrColors) { this.nrColors = nrColors; first = new Color(); Color previousColor = first; for (int i = 1; i < nrColors; i++) { final Color thisColor = new Color(); successor.put(previousColor, thisColor); previousColor = thisColor; } successor.put(previousColor, Color.none); } public Color firstColor() { return first; } boolean thereIsNextColor(Color color) { return successor.get(color) != Color.none; } public Color nextColor(Color color) { return successor.get(color); } } The structure we use is a Map. Map is an interface defined in the Java runtime and is available since the very early releases of Java. A Map has keys and value, and for any key, you can easily retrieve the value assigned to the key. You can see on the line, where the variable successor is defined that we define the type of the variable as an interface, but the value is an instance of a class. Obviously, the value cannot be an instance of an interface because such beasts do not exist. But, why do we define the variable to be an interface? The reason is abstraction and coding practice. If we need to change the implementation we use for some reason, the variable type still may remain the same and there is no need to change the code elsewhere. It is also a good practice to declare the variable to be an interface so that we will not have the temptation to use some special API of the implementation that is not available in the interface just by convenience. When it is really needed, we can change the type of the variable and use the special API. After all, there is a reason that API is there, but the mere temptation to use some special thing just because it is there is hindered. This helps to write simpler and cleaner program. is only one of the interfaces defined in the Java runtime belonging to the Java collections. There are many other interfaces and classes. Although, the JDK and all the classes are a vast amount and almost nobody knows all the classes that are there, collections is a special area that a professional developer should be knowledgeable about. Before getting into details on why HashMap is used in this code, we will have an overview of the collection classes and interfaces. This will help us also understand the other collections used in this program. Map Java collections Collections are interfaces and classes that help us store more than one object. We have already seen arrays that can do that, and also ArrayList in the previous chapters, but we did not discuss in detail what other possibilities there are in the JDK. Here, we will go into more detail, but leave the streams and the functional methods for later chapters, and we will also refrain to go into details that is rather the task of a reference book. Using implementation of the collection classes and interfaces reduces the programming effort. First of all, you do not need to program something that is already there. Secondly, these classes are highly optimized, both in implementation and in their features. They have very well designed API as well as the code is fast and uses small memory footprint. Sorry to say that their code was written long time ago and many times it is not a good style, hard to read, and understand. When you use a collection from the JDK, it is more likely that you can interoperate with some library. If you cook your own version of linked lists, it is not likely that you will find a readymade solution that will sort your list. If you use the LinkedList class in the JDK's standard class library, you will get a readymade solution from the Collections class, right from the JDK. It is also worth mentioning that the Java language itself supports these classes, for example, you can easily iterate through the elements of a Collection with a shortened special syntax. The collections in JDK contain interfaces that define the behavior of the different collection types, implementation classes, and algorithms that perform certain actions such as sorting. Many times, these algorithms work on different implementation versions, getting the same result, but optimized for the implementation specific class. You can use the API given by the interface, and if you change the implementation in your code, you will get an optimized version fitting the implementation. The collection interfaces can be categorized in two bags. One bag contains the interfaces that extend the Collection interface, and the other one contains Map, and a SortedMap extending Map. This way, Map is not really a collection, as it does not simply contain other objects but also pair values to keys. Interface collection Collection is the top of the interface hierarchy. This interface defines the methods that all implementations should provide, no matter if they implement the Set, SortedSet, List, Queue, or Deque interface directly. As Collection simply says that an object that implements the Collection interface is only an object that collects other objects together, the methods it defines are like adding a new object to the collection, clearing all elements from there, checking that an object is already a member of the collection, and iterating through the elements. For an up-to-date definition of the interface, consult the Java pi documentation (http://down load.java.net/java/jdk9/docs/api/overview-summary.html). You can consult the online API any time, and it is recommended to do so. The Java language itself directly supports the interface. You can iterate through the elements of the Collection with the enhanced for loop syntax, the same way as you can iterate over the elements of an array where the collection should be an expression that results an object that implements the Collection interface: for( E element : collection ){...} In the preceding code, E is either Object or the generic type of the elements of the Collection. The interface Collection is not directly implemented in the JDK. Classes implement one of the sub interfaces of Collection. Set The Set is a special collection that cannot contain duplicate elements. When you want to add an object into a set that already has an object that is the same or equal to the actual one, then the add method will not add the actual object. The add method will return false indicating the failure. You can use Set in your program when you need a collection of unique elements where you simply want to check that an element is a member of a set or not, whether an object belongs to a certain group or not. As we will return to our program code, we will see that the UniqueGuesser class has to implement an algorithm that checks that a color in a guess is present only once. This algorithm is the ideal candidate for a Set to be used: private boolean isNotUniqueWithSet(Color[] guess) { final Set<Color> alreadyPresent = new HashSet<>(); for (Color color : guess) { if (alreadyPresent.contains(color)) { return true; } alreadyPresent.add(color); } return false; } The code creates a set, which is empty when the method starts. After that, it checks for each color (notice the enhanced for loop over the array elements) if it was already present before. To do that, the code checks if the color is already in the set. If it is there, the guess is not unique as we have found a color that is present at least twice. If the color was not in the set, then the guess can still be unique in colors. To be able to detect that later, the code puts the color into the set. The actual implementation of Set that we will use is HashSet. In the JDK, there are many classes implementing the Set interface. The most widely used is HashSet, and it is also worth mentioning EnumSet, LinkedHashSet, and TreeSet. The last one also implements the SortedSet interface, so we will detail it there. To understand what HashSet (and later HashMap) are and how they work, we will have to discuss what hashes are. They play very important and central role in many applications. They do their job under the hood in the JDK but there are some very important constraints that programmers have to follow or else really weird and extremely hard to find bugs will make their life miserable. I dare to say that violation of the hash contract in HashSet and HashMap are the cause of the second most difficult to find bugs next to multithread issues. Thus, before going on with the different collection implementations, we will visit this topic. We are already one level deep from our example in this detour discussing collections and now we will go one level deeper. I promise this is the last in-depth level of detours. Hash functions A hash is a mathematical function that assigns a number to an element. Say you work at a university administration and you have to tell if Wilkinson is a student at your class. You can store the names on small papers in envelopes one for each starting letter. Instead of searching through the 10 thousand students, you can look at the papers in the envelope titled W. This very simple hash function assigns the first letter of the name to the name (or the ordinal number of the letter, as we said that a hash function results a number). This is not really a good hash function because it puts only a few elements, if any, into the envelope denoted X and many to A for example. A good hash function results each possible ordinal number with similar probability. In hash tables, we usually have more buckets (envelopes in the previous example) than the number of elements to be stored. Therefore, when an element is searched for, it is likely that there is only one element there. At least that is what we would like to have. If there are multiple elements in a single bucket, it is called collision. A good hash function has as little collisions as possible. For backward compatibility, there is a Hashtable class in the JDK. This was one of the first hash table implementations in Java right in the very first version, and as Java is backward compatible, it was not thrown away. The Map interface was introduced in version 1.2 only. Hashtable has many drawbacks and its use is not recommended. (Even the name is violating the Java naming conventions.) We do not discuss this class in this book. Whenever we talk about hash tables, it is referring to the actual array that is inside the implementation of HashSet, HashMap, or any other collection that uses some hash indexed table. Hash tables are arrays that use the result of the hash function to index the array. Usually, linked lists manage collisions. Hash table implementations also implement a strategy to resize the array when the number of elements to be stored becomes too high and the likelihood of collisions increase. This operation may take considerable time and, during this, the individual elements are moved between the buckets. During this operation, the hash table cannot reliably be used and this may be some source of issues in a multithread environment. In single thread code, you do not meet this problem. When you call the add method, the hash table (set or map) decides that the table has to be resized. The add method calls the resizing method and does not return until it is finished. Single thread code has no possibility to use the hash table during this period: the one and single thread is executing the resizing itself. In a multithread environment, however... and HashMap use the hash function provided by the Object that is stored in the collection. The Object class implements the hashCode and equals methods. You can override them and if you do, you should override both in a consistent manner. First, we will see what they are and then how to override them consistently. HashSet Method equals The documentation of set says "sets contain no pair of elements e1 and e2 such that e1.equals(e2)". The equals method returns true if the e1 and e2 are in some way equal. It may be different from two objects being identical. There can be two distinct objects that are equal. For example, we could have a color implementation that has the name of the colors as an attribute and two color objects may return true calling the equals method on one of them and passing the argument as the other when the two strings are equal. The default implementation of the equals method is in the code of the Object class and this returns true if and only if e1 and e2 are exactly the same and single object. It seems to be obvious, but my experience shows that it cannot be stressed enough that the implementation of equals in an object has to be as follows: Reflexive: This means that an object that always equals itself Symmetric (commutative): This means if e1.equals(e2) is true, then e2.equals(e1) should also be true Transitive: This means if e1.equals(e2) and e2.equals(e3), then e1.equals(e3) Consistent: This means that the return value should not change if the objects were not changed between the invocations Method hashCode The hashCode method returns an int. The documentation says that any class redefining this method should provide the following implementation: Consistently return the same value if the object was not modified Result the same int value for two objects that are equal (the equals method returns true) The documentation also mentions that this is not a requirement to result different int values for objects that are not equal, but it is desirable to support the performance of the hash implementing collections. If you violate any of these rules in the implementation of equals and hashCode, then the JDK classes using them will fail. As you can be sure that HashSet, HashMap, and similar classes were fully debugged, seeing that you added an object to a set and then the set reporting that it is not there will be a bewildering experience. However, only until you find out that the two objects being equal and stored in the set have different hashCode values, HashSet and HashMap will look for the object only in the bucket that is indexed by the hashCode value. It is also a common mistake to store an object in a HashSet or HashMap and then modify it. The object is in the collection but you cannot find it because the hashCode returns a different value. Objects stored in a collection should not be modified unless you know what you are doing. Many times, objects contain fields that are not interesting from the equality point of view. The hashCode and equals methods should be idempotent to those fields and you can alter those fields even after storing the object in a HashSet or in HashMap. As an example, you may administer triangles in objects maintaining the coordinates of the vertices and the color of the triangle. However, you do not care about the color for equality, only that the two triangles are at the exact same location in the space. In that case, the equals and hashCode method should not take the field color into account. This way, we can paint our triangles; they will still be found in HashSet or HashMap no matter what the color field is. Implementing equals and hashCode Implementing these methods is fairly simple. As this is a very common task, the IDEs support the generation of these methods. These methods are tied together so much that the menu items in the IDEs are not separate; they offer you to generate these methods at once. Asking the IDE to generate the equals method will result in something like the following code: @Override public boolean equals(Object o) { if (this == o) return true; if (o == null || getClass() != o.getClass()) return false; MyObjectJava7 that = (MyObjectJava7) o; return Objects.equals(field1, that.field1) && Objects.equals(field2, that.field2) && Objects.equals(field3, that.field3); } For this sample, we have three Object fields named field1, field2, and field3. The code with any other types and fields will look very similar. First, the method checks for object identity. One Object always equals itself. If the reference passed as argument is null and not an object, or they are of different class, then this generated method will return false. In other cases, the static method of the class Objects (note the plural) will be used to compare each of the fields. The utility class Objects was introduced in Java 7, hence the name of the sample class. The static methods, equals and hash, support the override of the Object equals and hashCode methods. The hashCode creation before Java 7 was fairly complex and required the implementation of modulo arithmetic with some magic numbers that is hard to explain just looking at the code without knowing the mathematics behind it. This complexity is now hidden behind the following Objects.hash method. @Override public int hashCode() { return Objects.hash(field1, field2, field3); } The generated method simply calls the Objects.hash method passing the important fields as arguments. HashSet Now, we know essentially a lot of things about hashes so we can bravely discuss the HashSet class. HashSet is an implementation of the Set interface that internally uses hash table. Generally, that is it. You store objects there and you can see if an object is already there or not. When there is a need for a Set implementation, almost always HashSet is the choice. Almost... EnumSet can contain elements from a certain enumeration. Recall that enumerations are classes that have fixed a number of instances declared inside the enum itself. As this limits the number of the different object instances, and this number is known during compilation time, the implementation of the EnumSet code is fairly optimized. Internally, EnumSet is implemented as a bit field and is a good choice where bit field manipulations can be used. EnumSet LinkedHashSet is a HashSet that also maintains a doubly linked list of the elements it holds. When we iterate though a HashSet, there is no guaranteed order of the element. When the HashSet is modified, the new elements are inserted into one of the buckets and, possibly, the hash table gets resized. This means that the elements get rearranged and get into totally different buckets. Iteration over the elements in HashSet just takes the buckets and the elements in it in some order that is arbitrary from the caller point of view. LinkedHashSet , however, iterates over the elements using the linked list it maintains and the iteration is guaranteed to happen in the order the elements were inserted. LinkedHashSet SortedSet The SortedSet is an interface that guarantees that the classes implementing it will iterate over the set in a sorted order. The order may be the natural ordering of the objects if the objects implement the Comparable interface or a Comparator object may drive it. This object should be available when the instance of the class implementing the SortedSet is created; in other words, it has to be a constructor parameter. NavigableSet extends the SortedSet interface with methods that let you do proximity search in the set. This essentially lets you search for an element that is in the search and is less than the searched object, less or equal to the searched element, greater or equal, or greater than the searched object. NavigableSet TreeSet is an implementation of NavigableSet and, this way this is also a SortedSet and, as a matter of fact, is also a Set. As a SortableSet documentation implies there are two types of the constructors, each having multiple versions though. One requires some Comparator, the other one relies on the natural ordering of the elements. TreeSet List is an interface that requires implementing class to keep track of the order of the elements. There are also methods that access an element by index and iteration defined by the Collection interface that guarantees the order of the elements. The interface also defines the listIterator method that returns an Iterator also implementing the ListIterator interface. This interface provides methods that let the caller insert elements to the list while iterating through it and also going back and forth in the iteration. It is also possible to search for a certain element in the List but most implementations of the interface provide poor performance while the searching is simply going through all elements until the element searched for is found. There are many classes implementing this interface in the JDK. Here, we will mention two. List LinkedList This is a doubly-linked list implementation of the List interface that has a reference to the previous, and also to the next element in the list for each element. The class also implements the Deque interface. It is fairly cheap to insert or delete an element from the list because it needs only the adjustment of few references. On the other hand, the access to an element by index will need iteration from the start of the list, or from the end of the list, whichever is closer to the specified indexed element. ArrayList This class is an implementation of the List interface that keeps the references to the elements in an array. That way, this is fairly fast to access an element by index. On the other hand, inserting an element to ArrayList can be costly. It needs moving all references above the inserted element one index higher, and it may also require resizing the backing array in case there is no room in the original one to store the new element. Essentially, this means allocating a new array and copying all references to it. The reallocation of the array may be optimized if we know how large the array will grow and call the ensureCapacity method. This will resize the array to the size provided as argument, even if the currently used slots are less numbered. My experience is that novice programmers use ArrayList when they need a list without considering the algorithmic performance of the different implementations. I do not actually know why there is this popularity of ArrayList. The actual implementation used in a program should be based on proper decision and not habit. Queue Queue is a collection that usually stores element for later use. You can put elements into a queue and you can pull them out. An implementation may specify the given order, that may be first in first out (FIFO) or last in first out (LIFO) or some priority based ordering. On a queue, you can invoke the add method to add an element, remove to remove the head element, and the element method to access the head element without removing it from the queue. The add method will throw an exception when there is a capacity problem and the element cannot be added to the queue. When the queue is empty, and there is no head element, the element and remove methods throw exception. As exceptions can only be used in exceptional cases, and the calling program may handle these situations in the normal course of the code, thus all these methods have a version that just return some special value signaling the situation. Instead of add, a caller may call offer to offer an element for storage. If the queue cannot store the element, it will return false. Similarly, peek will try to get access to the head element or return null if there is none, and poll will remove and return the head element or just return null if there is none. Note that these methods returning null just make the situation ambiguous when the implementation, such as LinkedList, allows null elements. Never store a null element in a queue. Deque is an interface which is a double-ended queue. It extends the Queue interface with the methods that allow access to both ends of the queue to add, look at, and remove elements from both ends. Deque For the Queue interface we needed six methods. Dequeue having two manageable ends needs 12 methods. Instead of add we have addFirst and addLast. Similarly we can offerFirst, offerLast as well as peekFirst, peekLast and pollFirst, pollLast. For some reason the methods that implement the functionality of the element method on Queue are named getFirst and getLast. Since this interface extends the Queue interface the methods defined there can also be used to access the head of the queue. In addition to these this interface also defines the methods removeFirstOccurrence and removeLastOccurrence that can be used to remove a specific element inside the queue. We cannot specify the index of the element to remove and we also cannot access an element based on index. The removeFirst/ LastOccurrence methods' argument is the object that is to be removed. If we need this functionality we can use Deque even if we add and remove elements from the same end of the queue. Why are there these methods in Deque and not in Queue? These methods have nothing to do with double headedness of Deque. The reason is that methods cannot be added to interfaces after they were released. If we add a method to an interface we break the backward compatibility because all classes that implement that interface have to implement the new method. Java 8 introduced default methods that eased this constraint, but the Queue interface was defined in Java 1.5 and the Deque interface was defined in Java 1.6. There was no way at that time to add the new methods to the already existing interfaces. Map A Map pairs keys and values. If we want to approach a Map from the Collection point of view then a Map is a set of key/value pairs. You can put key value pairs into a Map and you can get a value based on a key. Keys are unique the same way as elements in a Set. If you look at the source code of the different implementations of the Set interface, you may see that some of them are implemented as a wrapper around a Map implementation where the values are simply discarded. Using Maps is easy and alluring. Many languages, such as Python, Go, JavaScript, Perl, and so on, support this data structure on the language level. However, using a Map when an array would be sufficient is a bad practice that I have seen many times, especially in scripting languages. Java is not prone to that novice programmer error but you may still find yourself in a situation when you want to use a Map, and still there is a better solution. It is a general rule that the simplest data structure should be used that is sufficient for the implementation of the algorithm. HashMap is a hash table based implementation of the Map interface. As the map is based on a hash table, the basic put and get methods are performed in constant time. Additionally, as Map is very important, and because the most frequently used implementation in the JDK is HashMap, the implementation is fairly configurable. You can instantiate HashMap using the default constructor without argument, but there is also a constructor that defines the initial capacity and the load factor. HashMap IdentityHashMap is a special Map that implements the Map interface literally, but as a matter of fact, it violates the contract the Map interface documentation defines. It does it with good reason. The implementation uses a hash table just as HashMap, but to decide the equality of the key found in the bucket comparing with the key element provided as argument to the get method it uses Object reference (== operator) and not the method equals, which is required by documentation of Map interface. IdentityHashMap The use of this implementation is reasonable when we want to distinguish different Object instances as keys that otherwise equal to each other. Using this implementation for performance reasons is almost certainly a wrong decision. Also, note that there is no IdentityHashSet implementation in the JDK. Probably such collection is so rarely used that its existence in the JDK would cause more harm than good alluring novice programmers to misuse. Dependency injection In the previous chapter we briefly already discussed dependency injection (DI). Now we will dig into it a bit more detail. Objects usually do not work on their own. Most of the time the implementation depends on the services of other classes. When we want to write something to the console we use the System class. When we manage the table of guesses we need Color objects and ColorManager. In case of writing to the console we may not realize the dependency because the class being part of the JDK class library is available all the time and all we need to do is to write System.out.println. In this case this dependency is wired into the code. We cannot send the output somewhere else unless we change the code. This is not too flexible and in many cases we need a solution that can work with different output, different color manager or different whatever service our code depends on. The first step to do that is to have a field that has a reference of the object that gives our class the service. In case of output the type of the field can be of type OutputStream. The next, more interesting step is how this field gets value. One of the solution is to use DI. In this approach some external code prepares the dependencies and injects them into the object. When the first call to a method of the class is issued all the dependencies are already filled and ready to be used. In this structure, we have four different players: The client object is the one that gets the injected service objects during the process Service object or objects are injected into the client object Injector is the code that performs the injection Interfaces define the service that the client needs If we move the logic of the creation of the service objects from the client code the code becomes shorter and cleaner. The actual competency of the client class should hardly ever cover the creation of the service objects. For example a Game class contains a Table instance but a game is not responsible to create the Table. It is given to it to work with it, just as in real life that we model. The creation of service objects is sometimes as simple as issuing the new operator. Sometimes service objects also depend on other service objects and that way also act as clients in the process of dependency injection. In this case the creation of the service objects may be a lot of lines. The structure of the dependencies can be expressed in a declarative fashion that describes which service object needs which other service objects and also what implementation of the service interfaces are to be used. Dependency injection injectors work with such declarative descriptions. When there is a need for an object that needs service objects that themselves need again other service objects the injector creates the service instances in the appropriate order using the implementations that are matching the declarative descriptions. The injector discovers all the dependencies transitively and creates a transitive closure graph of the dependencies. The declarative description of the needed dependencies can be XML, or a special language developed especially for the dependency injection or it can even be Java itself using specially designed fluent API (h ttps://blog.jooq.org/2012/01/05/the-java-fluent-api-designer-crash-course/). XML was first used in DI injectors. Later Groovy based Domain Specific Language (https://martinfowler.com/books/dsl.html) came into picture and Java fluent API approach. We will use only the last one being the most modern and we will use Spring and GuiceDI containers since they are the most well-known injector implementations. Implementing the game Collections without examples are boring. Fortunately, we have our game where we use a few collection classes and also other aspects that we will examine in this chapter. ColorManager We jumped into the pool filled with collection classes from the implementation of the ColorManager class. Let's refresh the part of the class that is interesting for us now—the constructor: final protected int nrColors; final protected Map<Color, Color> successor = new HashMap<>(); final private Color first; public ColorManager(int nrColors) { this.nrColors = nrColors; first = new Color(); Color previousColor = first; for (int i = 1; i < nrColors; i++) { final Color thisColor = new Color(); successor.put(previousColor, thisColor); previousColor = thisColor; } successor.put(previousColor, Color.none); } We will use HashMap to keep the colors in an ordered list. At first, the choice of HashMap seems to be strange. Very true, that during the coding of ColorManager, I also considered a List, which seemed to be a more obvious choice. When we have a List<Color> colors variable, then the nextColor method is something like this: public Color nextColor(Color color) { if (color == Color.none) return null; else return colors.get(colors.indexOf(color) + 1); } The constructor will be much simpler, as shown in the following piece of code: final List<Color> colors = new ArrayList<>(); public ColorManager(int nrColors) { this.nrColors = nrColors; for (int i = 0; i < nrColors; i++) { colors.add(new Color()); } colors.add(Color.none); } public Color firstColor() { return colors.get(0); } Why did I choose the more complex solution and the unobvious data structure? The thing is performance. When the nextColor method is invoked, the list implementation first finds the element checking all the elements in the list and then fetches the next element. The time is proportional to the number of colors. When our number of colors increases, the time will also increase to just get the next color having one. At the same time, if we focus on not the data structure that comes from the verbal expression of the task we want to solve (get the colors in a sorted order) but rather focus on the actual method that we want to implement, nextColor(Color), then we will easily come to the conclusion that a Map is more reasonable. What we need is exactly a Map : having one element we want another related to the one we have. The key and the value is also Color. Getting the next element is constant time using HashMap. This implementation is probably faster than the one based on ArrayList. The problem is that it is only probably faster. When you consider refactoring a code to have better performance, your decision should always be based on measurements. If you implement a code that you only think is faster, practice shows, you will fail. In best case, you will optimize a code to be blazing fast and runs during the application server setup. At the same time, optimized code is usually less readable. Something for something. Optimization should never be done prematurely. Code for readability first. Then, assess the performance, and in case there is problem with the performance, then profile the execution and optimize the code where it hurts the most of the overall performance. Micro-optimizations will not help. Did I do premature optimization selecting the HashMap implementation instead of List? If I actually implemented the code using List and then refactored, then yes. If I was thinking about the List solution and then it came to me that Map solution is better without prior coding, then I did not. By years, such considerations will come easier, as you will also experience. The class color We have already looked at the code for the class code and it was the simplest class in the world. In reality, as it is in the GitHub repository (https://github.com/j9be/chapter04 or https://github.com/PacktPublishing/Java-9-Progra mming-By-Example/tree/master/Chapter04), the code is a bit more complex: package packt.java9.by.example.mastermind; /** * Represents a color in the MasterMind table. */ public class Color { /** * A special object that represents a * value that is not a valid color. */ public static final Color none = new Color(); } We have a special color constant named none that we use to signal a reference that is of type Color but is not a valid Color. In professional development, we used the null value for a long time to signal invalid reference, and because we are backward compatible, we still use it. However, it is recommended to avoid the null reference wherever possible. Tony Hoare (https://en.wikipedia.org/wiki/Tony_Hoare), who invented the null reference in 1965, admitted one time that this was a mistake that cost billions of dollars in the IT industry. The problem with the null value is that it takes the control away from the class, and thus, opens encapsulation. If a method returns null in some situation, the caller is strictly required to check the nullity and act according to that. For example, you cannot call a method on a null reference and you cannot access any field. If the method returns, a special instance of the object these problems are less serious. If the caller forgets to check the special return value and invokes methods on the special instance, the methods invoked still have the possibility to implement some exception or error handling. The class has the control encapsulated and can throw a special exception that may give more information about the error caused by the programmatic mistake by the caller not checking the special value. JavaDoc and code comments There is also another difference between what we presented here earlier and the listing. This is the commenting of the code. Code comments are part of the program, which are ignored, filtered out by the compiler. These comments are solely for those who maintain or use the code. In Java, there are two different comments. The code enclosed between /* and */ are comments. The start and the end of the comment do not need to be on the same line. The other type of comment starts with the // characters and ends at the end of the line. To document the code, the JavaDoc tool can be used. JavaDoc is a special tool that reads the source code and extracts HTML documentation about the classes, methods, fields, and other entities that have a comment starting with the /** characters. The documentation will contain the JavaDoc comments in a formatted way and also the information that is extracted from the program code. The documentation also appears as online help in the IDE when you move the mouse over a method call or class name, if there is any. The JavaDoc comment can contain HTML codes, but it generally should not. If really needed, you can use <p> to start a new paragraph or the <pre> tags to include some preformatted code sample into the documentation, but nothing more gives real benefit. Documentation should be as short as possible and contain as few formatting as possible. There are special tags that appear in the JavaDoc documentation. These are prefilled by the IDEs when you start to type a JavaDoc as /** and then press Enter. These are inside the comment and start with the @ character. There are a predefined set of tags: @author, @version, @param, @return, @exception, @see, @since, @serial, and @deprecated. The most important tags are @param and @return. They are used to describe the method arguments and the return value. Although we are not there yet, let's peek ahead to the guessMatch method from the Guesser class. /** * A guess matches if all rows in the table matches the guess. * * @param guess to match against the rows * @return true if all rows match */ protected boolean guessMatch(Color[] guess) { for (Row row : table.rows) { if (!row.guessMatches(guess)) { return false; } } return true; } The name of the parameter is automatically generated by the IDE. When you create the documentation, write something that is meaningful and not tautology. Many times, novice programmers feel the urge to write JavaDoc, and that something has to be written about the parameters. They create documentations like this: * @param guess is the guess Really? I would never have guessed. If you do not know what to write there to document the parameter, it may happen that you were choosing the name of the parameter excellent. The documentation of our preceding example will look as follows: Focus on what the method, class, and interface does and how it can be used. Do not explain how it works internally. JavaDoc is not the place for the explanation of the algorithm or the coding. It is used to help use the code. However, if somebody happens to explain how a method works, it is not a disaster. Comments can easily be deleted. There is, however, a comment that is worse than nothing: outdated documentation that is not valid anymore. When the contract of the element has changed, but the documentation does not follow the change and is misleading the user who wants to call the method, interface, or class whatever will face serious bugs and will be clueless. From now on, JavaDoc comments will not be listed in print to save trees, and electrons in the eBook version, but they are there in the repository and can be examined. Row Now, we have Colors and even instances if we need having a ColorManager. This is the time to store Colors in Rows. The Row class is a bit longer, but not too complex. package packt.java9.by.example.mastermind; import java.util.Arrays; public class Row { final Color[] positions; private int matchedPositions; private int matchedColors; A Row contains three fields. One is the positions array. Each element of the array is a Color. The matchedPositions is the number of positions that are matched and matchedColors is the number of colors that match a color in the hidden row but is not on the position as in the hidden row. public static final Row none = new Row(Guesser.none); The none is a constant that contains a special Row instance that we will use wherever we would use null. The constructor gets the colors in an array that should be in the row. public Row(Color[] positions) { this.positions = Arrays.copyOf(positions, positions.length); } The constructor makes a copy of the original array. This is an important code that we will examine a bit. Let's reiterate that Java passes arguments by value. It means that when you pass an array to a method, you will pass the value of the variable that holds the array. However, an array in Java is an Object just as well as anything else (except primitives like int). Therefore, what the variable contains is a reference to an object that happens to be an array. If you change the elements of the array, you actually change the elements of the original array. The array reference is copied when the argument passes, but the array itself, and the elements, are not. The java.util.Arrays utility class provides a lot of useful tools. We can easily code the array copying in Java but why to reinvent the wheel? In addition to that, arrays are continuous area of memory that can very effectively be copied from one place to another using low-level machine code. The copyOf method that we invoke calls the method System.arraycopy which is a native method and as such executes native code. Note that there is no guarantee that Arrays.copyOf invokes the native implementations and that this will be extremely fast in case of large arrays. The very version I was testing and debugging was doing it that way, and we can assume that a good JDK does something similar, effective and fast. After we copied the array, it is not a problem if the caller modifies the array that was passed to the constructor. The class will have a reference to a copy that will contain the same elements. However, note that if the caller changes any of the objects that are stored in the array (not the reference in the array, but the object itself that is referenced by an array element), then the same object is modified. Arrays.copyOf does not copy the objects that are referenced by the array, only the array elements. The row is created along with the colors and thus, we used a final field for the Color array. The matches, however, cannot be known when a Row is created. One of the players creates the Row and after that, the other player will tell the two int values. We do not create two setters for the two values, however, because they are always defined at the same time in the game together. public void setMatch(int matchedPositions, int matchedColors) { if (matchedColors + matchedPositions > positions.length) { throw new IllegalArgumentException( "Number of matches can not be more that the position."); } this.matchedColors = matchedColors; this.matchedPositions = matchedPositions; } The setMatch method does not only set the values, but also checks that the values are consistent. The sum of the two values cannot be more than the number of the columns. This check ensures that the caller, who uses the API of the Row class, does not use it inconsistently. If this API is used only from inside our code, this assertion should not be part of the code. A good coding style, in that case, will ensure that the method is never invoked inconsistently using unit tests. When we create API to use out of our control, we should check that the use is consistent. Failing to do so, our code may behave just weird when used inconsistently. When the caller sets matches to values that do not match any possible guess, the game may never finish and the caller may have a hard time figuring out what is going on. This figuring out probably will need the debug execution of our code. If we throw an exception in this case, the program stops where the bug is. There is no need to debug the library. public boolean guessMatches(Color[] guess) { return nrMatchingColors(guess) == matchedColors && nrMatchingPositions(guess) == matchedPositions; } The next method decides if a guess, given as an argument, matches the actual row. This method checks that the answers to the guess in the row can be valid if the current guess was in the hidden row. The implementation is fairly short and simple. A guess matches a row if the number of the colors matching and the number of positions matching are the same as the number given in the row. Do not be shy to write short methods. Do not think that a one-line method that essentially contains one statement is useless. Wherever we use this method, we could also write the expression, which is right after the return statement, but we do not for two reasons. The first and most important reason is that the algorithm, which decides that a row matches a guess belongs to the implementation of the class Row. If ever the implementation changes, the only location where the code is to be changed is here. The other reason is also important, and that is readability. In our codebase, we call this method from abstract class Guesser. It contains an if statement with the following expression: if (!row.guessMatches(guess)) { Would it be more readable in the following way: if( !(nrMatchingColors(guess) == matchedColors && nrMatchingPositions(guess) == matchedPositions)) { I am certain that the majority of the programmers understand the intention of the first version easier. I would even recommend implementing the doesNotMatchGuess method to improve the readability of the code even more. public int nrMatchingColors(Color[] guess) { int count = 0; for (int i = 0; i < guess.length; i++) { for (int j = 0; j < positions.length; j++) { if (i != j && guess[i] == positions[j]) { count++; } } } return count; } The number of matching colors is that which appears both in the row and the guess, but not in the same position. The definition, and how we calculate it, is fairly simple and unambiguous in case no color can appear twice in the hidden row. In case a color may appear multiple times in the hidden row, this implementation will count all occurrences of that color in the guess as many times as it appears in the hidden row. If we, for example, have a hidden RRGB row and the guess is bYRR, the calculation will say 4. It is a matter of agreement between the players how they count in this case. The important aspect is that they use the same algorithm, which should be true in our case, because we will ask the program to play both players. As we will program the code ourselves, we can trust that it will not cheat. public int nrMatchingPositions(Color[] guess) { int count = 0; for (int i = 0; i < guess.length; i++) { if (guess[i] == positions[i]) { count++; } } return count; } Counting the colors that are OK, and also on the position where they are supposed to be, is even simpler. public int nrOfColumns() { return positions.length; } This method tells the number of columns in the Row. This method is needed in the Game class that controls the flow of a whole game. As this class is in the same package as Row, it can access the field positions. I created the code to get the number of columns as row.positions.length. But then, I was reading the code next day and told myself: This is ugly and unreadable! What I am interested in here is not some mysterious positions' length; it is the number of columns. And the number of columns is the responsibility of the Row class and not the business of any other class. If I start to store the positions in a List, which does not have length (it has method size), it is the sole responsibility of Row and should not affect any other code. So, I created the nrOfColumns method to improve the code. The rest of the class contains some more very simple methods that are needed only to display the game and not for the algorithm to play: public int nrColumns() { return positions.length; } public Color position(int i) { return positions[i]; } public int matchedPositions() { return matchedPositions; } public int matchedColors() { return matchedColors; } } If you are a purist, you can encapsulate these methods into an inner class named Output or Print and call them through a final instance of it created as a field in the Row class. It is also possible to change the visibility of these fields from private to protected and implement these methods in a PrintableRow that can be instantiated from an already existing Row and implement these methods. The first version of PrintableRow will look like this: public class PrintableRow extends Row { public PrintableRow(Row row) { super(row.positions); super.setMatch(row.matchedPositions,row.matchedColors); } // the methods are deleted from the print ... } The methods are exactly the same as in the preceding print; they are cut and pasted, or rather moved, using the IDE refactoring support from one class to the other. When you write a code, please never use copy and paste. However you can use cut and paste to move code fragments around. The danger is in the copy paste use. Many developers claim that their use of actual copy and paste is not copy paste programming. Their reasoning is that they change the pasted code so much that it has practically nothing to do with the original code. Really? In that case why did you need the copied code when you started the modification of it? Why not start from scratch? That is because if you use the IDE's copy and paste functionality then, no matter what, you do copy paste programming. Class PrintableRow is pretty neat and separates the output concern from the core functionality. When you need an instance, it is not a problem that you have a Row instance already in hand. The constructor will essentially clone the original class and return a printable version. What bothers me is the implementation of the cloning. The code in the constructor calls the super constructor and then a method and all these work with the original functionality of the Row class. They have nothing to do with the printability that PrintableRow implements. This functionality actually belongs to the Row class. We should create a protected constructor that does the cloning: protected Row(Row cloneFrom) { this(cloneFrom.positions); setMatch(cloneFrom.matchedPositions, cloneFrom.matchedColors); } The constructor of PrintableRow should simply call super(row) and that is it. Code is never finished and never perfect. In a professional environment, programmers many times tend to finish polishing the code when it is good enough. There is no code that cannot be made better, but there is a deadline. The software has to be passed on to the testers and users and has to be used to help economy. After all, that is the final goal of a professional developer: have a code that supports the business. A code that never runs is worth nothing. I do not want you to think that the examples that I provided here were created perfect upfront. The reason for that is (did you read carefully?) because they are not perfect. As I said, code is never perfect. When I first created Row, it contained the printing methods in an inner class. I did not like it. The code was smelly. So, I decided to move the functionality to the Row class. However, I still did not like the solution. Then, I went to bed, slept, worked, and returned to it a few days later. What I could not create the day before now seemed obvious—these methods have to be moved to a subclass. Now comes another dilemma. Should I present this final solution or should I have here the different versions? In some cases, I will just present the final version. In other cases, like this, there are things to learn from the development step. In these cases, I present not only the code, but part of its evolution on how it was created. If you want to see those that I did not dare publishing, look at the Git history. I admit, sometimes, I create code that even makes me facepalm a day later. Table Table is a simple class that has only one very simple functionality. public class Table { final ColorManager manager; final int nrColumns; final List<Row> rows; public Table(int nrColumns, ColorManager manager) { this.nrColumns = nrColumns; this.rows = new LinkedList<>(); this.manager = manager; } public void addRow(Row row) { rows.add(row); } } There is one thing to mention, which is nothing new, but worth repeating. The rows variable is declared as final and it gets the value in the constructor. This is a List<Row> type variable. The fact that it is final means that it will hold the same list object during its lifetime. The length, members, and other features of the list may and will change. We will add new rows to this list. Final object variables reference an object, but it does not guarantee that the object itself is immutable. It is only the variable that does not change. When you do code review and explain to your colleagues what a class does, and you find yourself starting the explanation "this class is very simple" many times, it means the code is good. Well, it may be wrong in other aspects, but the class' granularity seems to be okay. Guesser Guesser and the UniqueGuesser and GeneralGuesser subclasses are the most interesting classes of the program. They actually perform the task that is the core of the game. Given a Table with a hidden row, the guesser has to create newer and newer guesses. To do this, a Guesser needs to get a Table when it is created. This is passed as a constructor argument. The only method it should implement is guess, which returns a new guess based on the table and on its actual state. As we want to implement a guesser that assumes that all colors in the hidden row are different, and also one that does not make this assumption, we will implement three classes. Guesser is an abstract class that implements only the logic that is independent from the assumptions. These methods will be inherited by both actual implementations: UniqueGuesser and GeneralGuesser. Let's go through the actual code of the class: package packt.java9.by.example.mastermind; public abstract class Guesser { protected final Table table; private final ColorManager manager; public Guesser(Table table) { this.table = table; this.lastGuess = new Color[table.nrColumns]; this.manager = table.manager; } The state of the guesser is the last guess it made. Although this is on the last row of the table, it is more of an internal matter of the guesser. The guesser has all the possible guesses, one after the other; lastGuess is the one where it left off last time and it should continue from there when it is invoked again. abstract protected void setFirstGuess(); Setting the first guess very much depends on the assumption of color uniqueness. The first guess should not contain duplicated colors in case the hidden row does not (at least in our implementation), while GeneralGuesser is free to guess any time, even as firstGuess all colors to be the same. protected final Color[] lastGuess; public static final Color[] none = new Color[]{Color.none}; Again, none in this class is just an object that we try to use instead of null, whenever we need to return something that is a reference to a Guess but is not really a guess. protected Color[] nextGuess() { if (lastGuess[0] == null) { setFirstGuess(); return lastGuess; } else { return nextNonFirstGuess(); } } The nextGuess method is an internal method that generates the next guess, which just comes as we order the possible guesses. It does not check anything against the Table; it only generates the next guess almost without thinking. The implementation on how we do the first guess and how we do the consecutive guesses are different. Thus, we will implement these algorithms in different methods and invoke them from here. The nextNonFirstGuess method represents the next guess in the special case when the guess is not the first one: private Color[] nextNonFirstGuess() { int i = 0; boolean guessFound = false; while (i < table.nrColumns && !guessFound) { if (manager.thereIsNextColor(lastGuess[i])) { lastGuess[i] = manager.nextColor(lastGuess[i]); guessFound = true; } else { lastGuess[i] = manager.firstColor(); i++; } } if (guessFound) { return lastGuess; } else { return none; } } Look back a few pages where we detailed how the algorithm works. We made the statement that this way of working is very much like the way we count with decimal numbers. By now, you have enough Java knowledge and programming skill to understand what the method does. It is more interesting to know why it is coded that way. Hint: as always, to be readable. There is the temptation to eliminate the guessFound variable. Would it not be simpler to return from the middle of the method when we find the blessed guesses? If we did, there would be no need to check the guessFound value before returning none value. The code would not get there if we returned from the middle of the loop. Yes, it would be simpler to write. But, we create code to be readable and not writable. Yes, but less code is more readable. Not in this case! Returning from a loop degrades the readability. Not to mention, the return statements are scattered around in the method at different stages of execution. private Color[] nextNonFirstGuess() { int i = 0; while (i < table.nrColumns) { if (manager.thereIsNextColor(lastGuess[i])) { lastGuess[i] = manager.nextColor(lastGuess[i]); return lastGuess; } else { lastGuess[i] = manager.firstColor(); i++; } } return none; } When somebody writes a code optimized in that way, it is similar to a toddler who makes his first steps and then looks proudly at the mother. Okay boy/girl, you are great. Now go on and start walking. When you are the postman, walking will be boring. That will be your profession. So, slide aside the pride and write boring code. Professionals write boring code. Won't it be slow? No! It will not be slow. First of all, it is not slow until the profiler proves that the code does not meet the business requirements. If it does, it is fast enough, no matter how slow it is. Slow is good as long as it is okay for the business. After all, JIT should have some task optimizing the code to run. The next method checks if the guess matches the previous guesses and their results on the Table: private boolean guessMatch(Color[] guess) { for (Row row : table.rows) { if (!row.guessMatches(guess)) { return false; } } return true; } private boolean guessDoesNotMatch(Color[] guess) { return !guessMatch(guess); } As we have the guess matching already implemented in the class Row, all we have to do is invoke that method for each row in the table. If all rows match, then the guess can be good for the table. If any of the former guesses do not match, then this guess goes down the drain. As we check the negated expression of matching, we created an English version of the method. In situations like this, it could be enough to create the guessDoesNotMatch version of the method. However, the logical execution of the code is more readable if the method is not negated. Therefore, it is more error prone to write the guessDoesNotMatch method alone. Instead, we will implement the original, readable version and the aux method to be nothing more than a negation. After all the aux methods, here we are implementing the public method of the Guesser. public Row guess() { Color[] guess = nextGuess(); while (guess != none && guessDoesNotMatch(guess)) { guess = nextGuess(); } if (guess == none) { return Row.none; } else { return new Row(guess); } } } It just takes the nextGuess and again and again until it finds one that matches the hidden row, or there is no more guess. If it finds a proper guess, it encapsulate it to a Row object and return it so that it can later be added to the Table by the Game objects. UniqueGuesser Class UniqueGuesser has to implement setFirstGuess (all concrete classes extending an abstract class should implement the abstract method of the parent) and it can and will override the protected nextGuess method: package packt.java9.by.example.mastermind; import java.util.HashSet; import java.util.Set; public class UniqueGuesser extends Guesser { public UniqueGuesser(Table table) { super(table); } @Override protected void setFirstGuess() { int i = lastGuess.length-1; for (Color color = table.manager.firstColor(); i >= 0; color = table.manager.nextColor(color)) { lastGuess[i--] = color; } } The setFirstGuess method selects the first guess in such a way that any possible color variations that come after the first one create the guesses one after the other if we follow the algorithm. The aux isNotUnique method returns true if the guess contains duplicate colors. It is not interesting to see how many. If all colors are the same, or only one color appears twice, it does not matter. The guess is not unique and does not fit our guesser. This method judges that. private boolean isNotUnique(Color[] guess) { final Set<Color> alreadyPresent = new HashSet<>(); for (Color color : guess) { if (alreadyPresent.contains(color)) { return true; } alreadyPresent.add(color); } return false; } To do this, it uses a Set, and any time a new color is found in the guess array, the color is stored in the set. If the set contains the color when we find it in the array, it means that the color was already used before; the guess is not unique. @Override protected Color[] nextGuess() { Color[] guess = super.nextGuess(); while (isNotUnique(guess)) { guess = super.nextGuess(); } return guess; } The overriding nextGuess method is simple. It asks the super nextGuess implementation to make guesses but throws away those that it does not like. package packt.java9.by.example.mastermind; <br/> <br/>public class GeneralGuesser extends Guesser { <br/> <br/> public GeneralGuesser(Table table) { super(table); } <br/> <br/> @Override <br/> protected void setFirstGuess() { <br/> int i = 0; <br/> for (Color color = table.manager.firstColor(); <br/> i < lastGuess.length; ) { <br/> lastGuess[i++] = color; <br/> } <br/> } <br/> <br/> } Setting the lastGuess it just puts the first color on all columns. Guess could not be simpler. Everything else is inherited from the abstract class Guesser. package packt.java9.by.example.mastermind; <br/> <br/> public class Game { <br/> <br/> final Table table; <br/> final private Row secretRow; <br/> boolean finished = false; <br/> <br/> public Game(Table table, Color[] secret ) { <br/> this.table = table; <br/> this.secretRow = new Row(secret); <br/> } <br/> <br/> public void addNewGuess(Row row) { <br/> if( isFinished()){ <br/> throw new IllegalArgumentException( <br/> "You can not guess on a finished game."); <br/> } <br/> final int positionMatch = secretRow. <br/> nrMatchingPositions(row.positions); <br/> final int colorMatch = secretRow. <br/> nrMatchingColors(row.positions); <br/> row.setMatch(positionMatch, colorMatch); <br/> table.addRow(row); <br/> if( positionMatch == row.nrOfColumns() ){ <br/> finished = true; <br/> } <br/> } <br/> <br/> public boolean isFinished() { <br/> return finished; <br/> } <br/> } Think about what I wrote earlier about short methods, and when you download the code from GitHub to play with it, try to make it look more readable. You can, perhaps, create and use a method named boolean itWasAWinningGuess(int positionMatch). Creating an integration test We have created unit tests in the previous chapter and there are unit tests for the functionalities implemented in the classes of this chapter as well. We will just not print these unit tests here. Instead of listing the unit tests, we will look at an integration test. Integration tests need the invocation of many classes working together. They check that the functionality can be delivered by the whole application, or at least a larger part of the application, and do not focus on a single unit. They are called integration tests because they test the integration between classes. The classes alone are all OK. They should not have any problem as it was already verified by the unit tests. Integration focuses on how they work together. If we want to test the Game class, we will either have to create mocks that mimic the behavior of the other Game classes, or we will just write an integration test. Technically, an integration test is very similar to a unit test. Many times, the very same JUnit framework is used to execute the integration tests. This is the case for the integration test of this game. The build tool, however, needs to be configured to execute the integration tests only when it is required. Usually, integration test executions need more time, and sometimes resources, such as external database that may not be available at each and every developer desktop. Unit tests run every time the application is compiled so they have to be fast. To separate the unit and integration tests, there are different techniques and configuration options, but there is no such more or less de-facto standard like the directory structure introduced by Maven (later adapted by Gradle). In our case, the integration test does not need any extra resource and does not take enormous time to run. It plays a game from the start to the end and plays the role of both the players. It is very much like somebody playing chess with themselves, making a step and then turning the table. The aim of this code is twofold. On one hand, we want to see that the code runs and plays a whole game. If the game finishes, then it is just OK. This is a very weak assertion and real integration tests perform lots of assertions (one test tests only one assertion though). We will focus on the other aim—deliver some joy and visualize the game on the console in text format so that the reader does not get bored. To do that, we will create a utility class that prints out a color and assigns letters to the Color instances on the fly. This is the PrettyPrintRow class. There are several limitations in this class that we have to talk about after we look at the code. I'd say that this code is here only to demonstrate what not to do, to establish some reasoning for the next chapter, and why we need to refactor the code we created in this one. package packt.java9.by.example.mastermind; import java.util.HashMap; import java.util.Map; public class PrettyPrintRow { private static final Map<Color, Character> letterMapping = new HashMap<>(); private static final String letters = "RGBYWb"; private static int counter = 0; private static char colorToChar(Color color) { if (!letterMapping.containsKey(color)) { letterMapping.put(color, letters.charAt(counter)); counter++; } return letterMapping.get(color); } This is the heart of this class. When a color is to be printed, it gets a letter assigned unless it already has one. As the Map containing the assignments in each and every game that is running in the JVM will use the same mapping, a new Game is started. It allocates new Colors and will soon run out of the six characters that we allocated here in the String constant. If the Game instances are run parallel, then we are in even more trouble. The class is not thread safe at all. If two threads concurrently call the colorToChar method for the same Color instance, (which is not likely because each Game uses its own color, but note that not likely in programming is very much like a famous last words quote on a tombstone) then both threads may see at the same time that there is no letter assigned to the color and both will assign the letter (the same letter or two different letters, based on luck) and increase the counter once or twice. At least, what we can say is that the execution is nondeterministic. You may recall that I said violating the hash contract is the second most difficult to find bug after multithread issues. Such a nondeterministic code is exactly that: a multithread issue. There is no prize to find the most difficult bug. When the application does not run, and a bug affects the production system for hours or days, no businessperson will be happy, and they will not be amazed after you find the bug. It may be an intellectual challenge, but the real value is not creating the bugs in the first place. As a summary, this code can only be used once in a JVM by a single thread. For this chapter, it is good, though a smelly and shameful code, but it will be a good example for the next chapter, in which we will see, how to refactor the application so that it will not need such a hacking to print out the colors. Code smell is a term minted by Kent Back, according to Martin Fowler (http://martinfowler.co m/bliki/CodeSmell.html). It means that some code looks not good, nor apparently bad, but some constructs make the feeling in the developer that it may not be good. As it is defined on the web page, "A code smell is a surface indication that usually corresponds to a deeper problem in the system." The term is widely accepted and used in software development for the last 10 years. The rest of the code is plain and simple: public static String pprint(Row row) { String string = ""; PrintableRow pRow = new PrintableRow(row); for (int i = 0; i < pRow.nrOfColumns(); i++) { string += colorToChar(pRow.position(i)); } string += " "; string += pRow.matchedPositions(); string += "/"; string += pRow.matchedColors(); return string; } The integration test, or rather the demonstration code (as it does not contain any assertions other than it runs without exception), defines six colors and four columns. This is the size of the original game. It creates a color manager, and then it creates a table and a secret. The secret could be just any random color selection from the six colors that is available (there are 360 different possibilities tested in the UniqueGuesserTest unit test available from GitHub). As we know that the Guesser implementation starts from one end of the color set and creates the new guesses systematically, we want to set a secret that it will guess the last. This is not because we are evil, but rather because we want to see that our code really works. The directory structure of the code is very similar to the one we used in case of the Maven build tool, as can be seen on the following screenshot created on a Windows machine: The source code is under the directory src and the main and test source code files are separated into two subdirectory structures. The compiled files will be generated in the directory build when we use Gradle. The code of the integration test class is the following: package packt.java9.by.example.mastermind.integration; import org.junit.Assert; import org.junit.Test; import packt.java9.by.example.mastermind.*; public class IntegrationTest { final int nrColors = 6; final int nrColumns = 4; final ColorManager manager = new ColorManager(nrColors); private Color[] createSecret() { Color[] secret = new Color[nrColumns]; int count = 0; Color color = manager.firstColor(); while (count < nrColors - nrColumns) { color = manager.nextColor(color); count++; } for (int i = 0; i < nrColumns; i++) { secret[i] = color; color = manager.nextColor(color); } return secret; } @Test public void testSimpleGame() { Table table = new Table(nrColumns, manager); Color[] secret = createSecret(); System.out.println( PrettyPrintRow.pprint(new Row(secret))); System.out.println(); Game game = new Game(table, secret); Guesser guesser = new UniqueGuesser(table); while (!game.isFinished()) { Row guess = guesser.guess(); if (guess == Row.none) { Assert.fail(); } game.addNewGuess(guess); System.out.println(PrettyPrintRow.pprint(guess)); } } } The easiest way to run the test is start it from inside the IDE. When the IDE imports the project based on the build file, be it a Maven pom.xml or Gradle build.gradle. IDE usually provides a run button or menu to start the code. Running the game will print out the following piece of code that we worked so hard on in this chapter: RGBY GRWb YBbW BYGR RGYB RGBY 0/0 0/2 0/2 0/4 2/2 4/0 Summary In this chapter, we programmed a table game: Mastermind. We not only programmed the model of the game, but also created an algorithm that can guess. We revisited some OO principles and discussed why the model was created the way it was. While we created the model of the game, which we will refine in the next chapter, you have learned about Java collections, what an integration test is, and how to create JavaDoc. Extending the Game - Run Parallel, Run Faster In this chapter, we will extend the Mastermind game. As it is now, it can guess the secret that was hidden and also hide the pegs. The test code can even do both at the same time. It can play against itself leaving us only with the fun of programming. What it cannot do is make use of all the processors that we have in today's notebooks and servers. The code runs synchronous and utilizes only a single processor core. We will alter the code extending the guessing algorithm to slice up the guessing into subtasks and execute the code in parallel. During this, we will get acquainted with Java concurrent programming. This will be a huge topic with many subtle corners and caveats lurking in the dark. We will get into those details that are the most important and will form a firm base for further studies whenever you need concurrent programs. As the outcome of the game is the same as it was, only faster, we have to assess what faster is. To do that, we will utilize a new feature introduced in Java 9: microbenchmarking harness. In this chapter, we will cover the following topics: The meaning of processes, threads and fibers Multithreading in Java Issues with multithread programming and how to avoid them Locking, synchronization, and blocking queues Microbenchmarking How to make Mastermind parallel The old algorithm was to go through all the variations and try to find a guess that matches the current state of the table. Assuming that the currently examined guess is the secret, will we get the same answers for the guesses that are already on the table as the answers are actually on the table? If yes, then the current guess can be the secret, and it is just as good a guess as any other guesses. A more complex approach can implement the min-max algorithm (https://en.wikipedia.org/wiki/Minimax). This algorithm does not simply get the next possible guess but also looks at all the possible guesses and selects the one that shortens the outcome of the game the most. If there is a guess that can be followed by three more guesses in the worst case, and there is another for which this number is only two, then min-max will choose the latter. It is a good exercise for the interested readers. In the case of the six colors and four columns for the pegs, the min-max algorithm solves the game in no more than 5 steps. The simple algorithm we implemented also solves the game in 5 steps. However, we do not go in that direction. Instead, we want to have a version of the game that utilizes more than one processor. How can you transform the algorithm into a parallel one? There is no simple answer to this. When you have an algorithm, you can analyze the calculations and parts of the algorithm, and you can try to find dependencies. If there is some calculation B that needs the data, which is the result of another calculation A, then it is obvious that A can only be performed when B is ready. If there are parts of the algorithm that do not depend on the outcome of the others, then they can be executed in parallel. For example, the quick-sort has two major tasks: partitioning and then sorting of the two parts. It is fairly obvious that the partitioning has to finish before we start sorting the two partitioned parts. However, the sorting tasks of the two parts do not depend on each other, they can be done independently. You can give them to two different processors. One will be happy sorting the part containing the smaller elements; the other one will carry the heavier, larger ones. If you turn the pages back to Chapter 3, Optimizing the Sort - Making Code Professional where we implemented quick-sort in a non-recursive way, you can see that we scheduled sorting tasks into a stack and then performed the sorting by fetching the elements from the stack in a while loop. Instead of executing the sort right there in the core of the loop, we could pass the task to an asynchronous thread to perform it and go back for the next waiting task. We just do not know how. Yet. That is why we are here in this chapter. Processors, threads, and processes are complex and abstract things and they are hard to imagine. Different programmers have different techniques to imagine parallel processing and algorithms. I can tell you how I do it but it is not a guarantee that this will work for you. Others may have different techniques in their mind. As a matter of fact, I just realized that as I write this, I have actually never told this to anyone before. It may seem childish, but anyway, here it goes. When I imagine algorithms, I imagine people. One processor is one person. This helps me overcome the freaking fact that a processor can make billions of calculations in a second. I actually imagine a bureaucrat wearing a brown suit and doing the calculations. When I create a code for a parallel algorithm, I imagine many of them working behind their desks. They work alone and they do not talk. It is important that they do not talk to each other. They are very formal. When there is a need for information exchange, they stand up with a piece of paper they have written something on, and they bring it to each other. Sometimes, they need a piece of paper for their work. Then they stand up, go to the place where the paper is, take it, bring it back to their desk, and go on working. When they are ready, they go back and bring the paper back. If the paper is not there when they need it, they queue up and wait until someone who has the paper brings it there. How does it help with Mastermind? I imagine a boss who is responsible for the guesses. There is a table on the wall in the office with the previous guesses and the results for each row. The boss is too lazy to come up with new guesses so he gives this task to subordinates. When a subordinate comes up with a guess, the boss checks whether the guess is valid or not. He does not trust the subordinates, and if the guess is good, he makes it as an official guess, putting it on the table along with the result. The subordinates deliver the guesses written on small post-it notes, and they put them in a box on the table of the boss. The boss looks at the box from time to time, and if there is a note, the boss takes it. If the box is full and a subordinate wants to put a paper there, the subordinate stops and waits until the boss takes at least one note so that there is some room in the box for a new note. If the subordinates queue up to deposit guesses in the box, they all wait for their time. The subordinates should be coordinated; otherwise, they will just come up with the same guesses. Each of them should have an interval of guesses. For example, the first one should check the guesses from 1234 up until 2134, the second should check from 2134 up until 3124, and so on, if we denote the colors with numbers. Will this structure work? Common sense says that it will. However, bureaucrats, in this case, are metaphors and metaphors are not exact. Bureaucrats are human, even when they do not seem like it much more than threads or processors. They sometimes behave extremely strangely, doing things that normal humans don't really do often. However, we can still use this metaphor if it helps us imagine how parallel algorithms work. We can imagine that the boss goes on holiday and does not touch the heap of paper piling up on the table. We can imagine that some of the workers are producing results much faster than the others. As this is only imagination, the speedup can be 1000 times (think of a time-lapse video). Imagining these situations may help us discover special behavior that rarely happens, but may cause problems. As the threads work in parallel, many times subtle differences may influence the general behavior greatly. In some early version, as I coded the parallel Mastermind algorithm, the bureaucrats started working and filled the box of the boss with guesses before the boss could put any of them on the table. As there were no guesses on the table, the bureaucrats simply found all possible variations in their interval being a possibly good guess. The boss gained nothing by the help of the parallel helpers; they had to select the correct ones from all possible guesses, while the guessers were just idle. Another time, the bureaucrats were checking guesses against the table while the boss was putting a guess on one of them created beforehand. And some of the bureaucrats freaked out saying that it is not possible to check a guess against a table if someone is changing it. More precisely, the code executing in one thread, threw ConcurrentModificationException when the List of the table was modified. Another time, I tried to avoid the too fast work of bureaucrats, and I limited the size of the box where they could put their papers containing the guesses. When the boss finally found the secret, and the game finished, the boss told the bureaucrats that they could go home. The boss did that by creating a small paper with the instruction: you can go home, and put it on the tables of the bureaucrats. What did the bureaucrats do? Kept waiting for the box to have space for the paper! (Until the process was killed. This is kind of equivalent on Mac OS and on Linux as ending the process from the task manager on Windows.) Such coding errors happen and, to avoid as many as possible, we have to do at least two things. Firstly, we have to understand how Java multithreading works and secondly, have a code as clean as possible. For the second, we will clean up the code even more and then we will look at how the parallel algorithm described earlier can be implemented in Java, running on the JVM instead of utilizing bureaucrats. Refactoring When we finished the previous chapter, we had the classes of the Mastermind game designed and coded in a nice and perfectly object oriented way that did not break any of the OO principles. Did we? Absurd. There is no code, except some trivial examples, that cannot be made to look nicer or better. Usually, when we develop code and finish the coding, it looks great. It works, the tests all run, and documentation is ready. From the professional point of view, it really is perfect. Well, it is good enough. The big question that we have not tested yet is maintainability. What is the cost to alter the code? That is not an easy question, especially because it is not a definite one. Alter to what? What is the modification that is to be made to the code? We do not know that when we create the code in the first place. If the modification is to fix a bug, then it is obvious that we did not know that beforehand. If we knew, we would not have introduced the bug in the first place. If this is a new feature, then there is a possibility that the function was foreseen. However, usually it is not the case. When a developer tries to predict the future, and what features the program will need in the future, they usually fail. It is the task of the customer to know the business. Features needed are driven by the business in case of professional software development. After all, that is what it means to be professional. Even though we do not exactly know what needs to be altered later in the code, there are certain things that may give hints to experienced software developers. Usually, the OO code is easier to maintain than the ad-hoc code, and there are code smells that one can spot. For example, take a look at the following code lines: while (guesser.guess() != Row.none) { while (guesser.nextGuess() != Guesser.none) { public void addNewGuess(Row row) { Color[] guess = super.nextGuess(); We may sense the odor of something strange. (Each of these lines is in the code of the application as we finished it in Chapter 4, Mastermind - Creating a Game.) The return value of the guess method is compared to Row.none, which is a Row. Then, we compare the return value of nextGuess to Guesser.none, which should be a Guesser. When we add a new guess to something, we actually add a Row. Finally, we can realize that nextGuess returns a guess that is not an object with its own declared class. A guess is just an array of colors. Should we introduce another layer of abstraction creating a Guess class? Will it make the code more maintainable? Or will it only make the code more complex? It is usually true that the less code lines we have, the less possibility we have for bugs. However, sometimes, the lack of abstraction will make the code complex and tangled. What is the case in this situation? How can we decide that generally? The more experience you have, the easier you will tell by looking at the code and acutely knowing what modifications you want to make. Many times, you will not bother making the code more abstract, and many other times, you will create new classes without hesitation. When in doubt, do create the new classes and see what comes out. The important thing is not to ruin the already existing functionality. You can do that only if you have sufficient unit tests. When you want to introduce some new functionality or fix a bug, but the code is not appropriate, you will have to modify it first. When you modify the code so that the functionality does not change, the process is named refactoring. You change a small part of the code in a limited time, and then you build it. If it compiles and all unit tests run, then you can go on. The hint is to run the build frequently. It is like building a new road near an existing one. Once in every few miles, you should meet the old line. Failing to do so, you will end up somewhere in the middle of the desert in a totally wrong direction, and all you can do is return to the starting point—your old to-be-refactored code. Effort wasted. It is not only the safety that advises us to run the build frequently, it is also time limitation. Refactoring does not directly deliver revenue. The functionality of the program is tied directly to income. Nobody will pay us for infinite refactoring work. Refactoring has to stop some time and it is usually not the time when there is nothing to be refactored any more. The code will never be perfect, but you may stop when it is good enough. And, many times, programmers are never satisfied with the quality of the code, and when they are stopped by some external factor (usually called project manager), the code should compile and tests should run so that the new feature and bug fixing can be performed on the actual code base. Refactoring is a huge topic and there are many techniques that can be followed during such an activity. It is so complex that there is a whole book about it by Martin Fowler (http://martinfowler.com/books/refactoring.html). In our case, the modification we want to apply to our code is to implement a parallel algorithm. The first thing we will modify is the ColorManager. When we wanted to print guesses and rows on the terminal, we had to implement some bad tricks. Why not have color implementations that can be printed? We can have a class that extends the original Color class and has a method that returns something that represents that color. Do you have any candidate name for that method? It is the toString method. It is implemented in the Object class and any class can freely override it. When you concatenate an object to a string, automatic type conversion will call this method to convert the object to String. By the way, it is an old trick to use ""+object instead of object.toString() to avoid null pointer exception. Needless to say, we do not use tricks. The toString method is also invoked by the IDEs when the debugger wants to display the value of some object, so it is generally recommended to implement toString if for nothing else, then to ease development. If we have a Color class that implements toString, then the PrettyPrintRow class becomes fairly straightforward and tricks less: package packt.java9.by.example.mastermind; public class PrettyPrintRow { public static String pprint(Row row) { String string = ""; PrintableRow pRow = new PrintableRow(row); for (int i = 0; i < pRow.nrOfColumns(); i++) { string += pRow.pos(i); } string += " "; string += pRow.full(); string += "/"; string += pRow.partial(); return string; } } We removed the problem from the printing class, but you may argue that the issue is still there, and you are right. Many times, when there is a problem in a class design, the way to the solution to move the problem from the class to another. If it is still a problem there, then you may split the design more and more and, at the last stage, you will realize that what you have is an issue and not a problem. To implement a LetteredColor class is also straightforward: package packt.java9.by.example.mastermind.lettered; import packt.java9.by.example.mastermind.Color; public class LetteredColor extends Color { private final String letter; public LetteredColor(String letter){ this.letter = letter; } @Override public String toString(){ return letter; } } Again, the problem was pushed forward. But, in reality, this is not a problem. It is an OO design. Printing is not responsible for assigning String to colors for their representation. And the color implementation itself is also not responsible for that. The assignment has to be performed where the color is made, and then the String has to be passed to the constructor of the LetteredColor class. The color instances are created in ColorManager so we have to implement this in the ColorManager class. Or not? What does ColorManager do? It creates the colors and... When you come to an explanation or description of a class that lists the functionalities, you may immediately see that the single responsibility principle was ignored. ColorManager should manage the colors. Managing is providing a way to get the colors in a definite order and getting the first and the next when we know one color. We should implement the other responsibility—the creation of a color in a separate class. A class that has the sole functionality to create an instance of another class is called factory. That is almost the same as using the new operator but unlike new, the factories can be used more flexibly. We will see that immediately. The ColorFactory interface contains a single method, as follows: package packt.java9.by.example.mastermind; public interface ColorFactory { Color newColor(); } Interfaces that define only one method are named functional interfaces because their implementation can be provided as a lambda expression at the place where you would use an object that is an instance of a class which implements the functional interface. The SimpleColorFactory implementation creates the following Color objects: package packt.java9.by.example.mastermind; public class SimpleColorFactory implements ColorFactory { @Override public Color newColor() { return new Color(); } } It is very much like how we create an interface, and then an implementation, instead of just writing new Color() in the code in ColorManager. LetteredColorFactory is a bit more interesting: package packt.java9.by.example.mastermind.lettered; import packt.java9.by.example.mastermind.Color; import packt.java9.by.example.mastermind.ColorFactory; public class LetteredColorFactory implements ColorFactory { private final String letters = "0123456789ABCDEFGHIJKLMNOPQRSTVWXYZabcdefghijklmnopqrstvwxzy"; private int counter = 0; @Override public Color newColor() { Color color = new LetteredColor(letters.substring(counter, counter + 1)); counter++; return color; } } Now, here we have the functionality that assigns Strings to the Color objects when they are created. It is very important that the counter variable that keeps track of the already created colors is not static. The similar variable in the previous chapter was static and it meant that it could run out of characters as evernewer ColorManagers created too many colors. It actually did happen to me during the unit test execution when the tests each created ColorManagers and new Color instances, and the printing code tried to assign new letters to the new colors. The tests were running in the same JVM under the same classloader and the unfortunate static variable had no clue when it could just start counting from zero for the new tests. The drawback is that somebody, somewhere, has to instantiate the factory and it is not the ColorManager. ColorManager already has a responsibility and it is not to create a color factory. The ColorManager has to get the ColorFactory in its constructor: package packt.java9.by.example.mastermind; import java.util.HashMap; import java.util.List; import java.util.Map; public class ColorManager { final protected int nrColors; final protected Map<Color, Color> successor = new HashMap<>(); private Color first; private final ColorFactory factory; public ColorManager(int nrColors, ColorFactory factory) { this.nrColors = nrColors; this.factory = factory; createOrdering(); } private Color[] createColors() { Color[] colors = new Color[nrColors]; for (int i = 0; i < colors.length; i++) { colors[i] = factory.newColor(); } return colors; } private void createOrdering() { Color[] colors = createColors(); first = colors[0]; for (int i = 0; i < nrColors - 1; i++) { successor.put(colors[i], colors[i + 1]); } } public Color firstColor() { return first; } public boolean thereIsNextColor(Color color) { return successor.containsKey(color); } public Color nextColor(Color color) { return successor.get(color); } public int getNrColors() { return nrColors; } } You may also notice that I could not resist refactoring the createColors method into two methods to follow the single responsibility principle. Now, the code that creates ColorManager has to create a factory and pass it to the constructor. For example, the unit test's ColorManagerTest class will contain the following method: @Test public void thereIsAFirstColor() { ColorManager manager = new ColorManager(NR_COLORS, Color::new); Assert.assertNotNull(manager.firstColor()); } This is the simplest way ever to implement a factory defined by a functional interface. Just name the class and reference the new operator like it was a method by creating a method reference. The next thing we will refactor is the Guess class, which, actually, we did not have so far. A Guess class contains the pegs of the guess and can calculate the number of full (color as well as position) and partial (color present but in wrong position) matches, and can also calculate the next Guess that comes after this guess. This functionality was implemented in the Guesser class so far, but this is not really the functionality for how we select the guesses when checking the already made guesses on the table. If we follow the pattern we set up for the colors, we may implement this functionality in a separate class named GuessManager, but as for now, it is not needed. Again, this is not black and white. It is important to note that a Guess object can only be made at once. If it is on the table, the player is not allowed to change it. If we have a Guess that is not yet on the table, it is still just a Guess identified by the colors and orders of the pegs. A Guess object never changes after it was created. Such objects are easy to use in multithread programs and are called immutable objects: package packt.java9.by.example.mastermind; import java.util.Arrays; import java.util.HashSet; import java.util.Set; public class Guess { final static public Guess none = new Guess(new Color[0]); final private Color[] colors; private boolean uniquenessWasNotCalculated = true; private boolean unique; public Guess(Color[] colors) { this.colors = Arrays.copyOf(colors, colors.length); } The constructor is creating a copy of the array of colors that were passed. As a Guess is immutable, this is extremely important. If we just keep the original array, any code outside of the Guess class could alter the elements of the array, essentially changing the content of Guess that is not supposed to be changing: public Color getColor(int i) { return colors[i]; } public int nrOfColumns() { return colors.length; } /** * Calculate the next guess and return a new Guess object. * The guesses are ordered in the order of the colors as * specified by the color manager. * * @param manager that specifies the order of the colors * can return the next color after one color. * @return the guess that comes after this guess. */ public Guess nextGuess(ColorManager manager) { final Color[] colors = Arrays.copyOf( this.colors, nrOfColumns()); int i = 0; boolean guessFound = false; while (i < colors.length && !guessFound) { if (manager.thereIsNextColor(getColor(i))) { colors[i] = manager.nextColor(colors[i]); guessFound = true; } else { colors[i] = manager.firstColor(); i++; } } if (guessFound) { return new Guess(colors); } else { return Guess.none; } } In this method, we start to calculate the next Guess starting with the color array that is contained in the actual object. We need a work array that is modified, so we will copy the original. The final new object can, this time, use the array we use during the calculation, so that will need a separate constructor that does not create a copy. It is possible extra code, but we should consider making that only if we see that that is the bottleneck in the code and we are not satisfied with the actual performance. The next method just checks if the passed Guess has the same number of colors as the actual one. This is just a safety check used by the next two methods that calculate the matches: private void assertCompatibility(Guess guess) { if (nrOfColumns() != guess.nrOfColumns()) { throw new IllegalArgumentException("Cannot compare different length guesses"); } } /** * Count the number of colors that are present on the guess * but not on the pos where they are in the other guess. * If the same color is on multiple pos it is counted * for each pos once. For example the secret is * <pre> * RGRB * </pre> * and the guess is * <pre> * YRPR * </pre> * then this method will return 2. * * @param guess is the actual guess that we evaluate * @return the number of good colors not in pos */ public int nrOfPartialMatches(Guess guess) { assertCompatibility(guess); int count = 0; for (int i = 0; i < nrOfColumns(); i++) { for (int j = 0; j < nrOfColumns(); j++) { if (i != j && guess.getColor(i) == this.getColor(j)) { count++; } } } return count; } /** * Count the number of colors that are correct and are in pos. * * @param guess is the actual guess that we evaluate * @return the number of colors that match in pos */ public int nrOfFullMatches(Guess guess) { assertCompatibility(guess); int count = 0; for (int i = 0; i < nrOfColumns(); i++) { if (guess.getColor(i) == this.getColor(i)) { count++; } } return count; } The isUnique method checks if there is any color more than once in the Guess. As the Guess is immutable, it may not happen that a Guess is unique one time and not unique at another time. This method should return the same result whenever it is called on a specific object. Because of that, it is possible to cache the result. This method does this, saving the return value to an instance variable. You may say that this is premature optimization. Yes, it is. I decided to do it for one reason. It is demonstration, and based on that, you can try to modify the nextGuess method to do the same: /** * @return true if the guess does not * contain any color more than once */ public boolean isUnique() { if (uniquenessWasNotCalculated) { final Set<Color> alreadyPresent = new HashSet<>(); unique = true; for (Color color : colors) { if (alreadyPresent.contains(color)) { unique = false; break; } alreadyPresent.add(color); } uniquenessWasNotCalculated = false; } return unique; } Methods that return the same result for the same arguments are called idempotent. Caching the return value for such a method can be very important if the method is called many times and the calculation is using a lot of resources. When the method has arguments, the result caching is not simple. The object method has to remember the result for all arguments that were already calculated, and this storage has to be effective. If it takes more resources to find the stored result than the calculation of it, then the use of cache not only uses more memory but also slows down the program. If the method is called for several arguments during the lifetime of the object, then the storage memory may just grow too large. Some of the elements have to be purged—those that will not be needed anymore in the future. However, we cannot know which elements of the cache are not needed, so we will have to guess. As you can see, caching can get complex very fast and, to do that professionally, it is almost always better to use some readily available cache implementation. The caching we use here is only the tip of the iceberg. Or, it is even only the sunshine glimpsing on it. The rest of the class is fairly standard and something we have talked about in detail—a good check of your knowledge is to understand how the equals, hashCode, and toString methods are implemented this way. I implemented the toString method to help me during debugging, but it is also used in the following example output: @Override public boolean equals(Object other) { if (this == other) return true; if (other == null || !(other instanceof Guess)) return false; Guess guess = (Guess) other; return Arrays.equals(colors, guess.colors); } @Override public int hashCode() { return Arrays.hashCode(colors); } @Override public String toString() { if (this == none) { return "none"; } else { String s = ""; for (int i = colors.length - 1; i >= 0; i--) { s += colors[i]; } return s; } } } This is mainly the modification that I needed while I developed the parallel algorithm. Now, the code is fairly up-to-date and described to focus on the main topic of this chapter: how to execute code in Java in parallel. The parallel execution of the code in Java is done in threads. You may know that there is a Thread object in Java runtime, but without understanding what a thread in the computer is, it makes no sense. In the following subsections, we will learn what these threads are, how to start a new thread, how to synchronize data exchange between threads, and finally put all this together and implement the Mastermind parallel guessing algorithm. Processes When you start your computer, the program that starts is the operating system (OS). The OS controls the machine hardware and the programs that you can run on the machine. When you start a program, the OS creates a new process. It means that the OS allocates a new entry in a table (array) where it administers the processes and fills in the parameters that it knows, and needs to know, about the process. For example, it registers what memory segment the process is allowed to use, what the ID of the process is, and which user started from which other process. You cannot start a process just out of thin air. When you double-click on an EXE file, you actually tell the file explorer, which is a program running as a process, to start the EXE file as a separate process. The explorer calls the system via some API and kindly asks the OS to do that. The OS will register the explorer process as the parent of the new process. The OS does not actually start the process, but creates all the data that it needs to start it and, when there is some free CPU resource, then the process gets started, and then it gets paused very soon. You will not notice it because the OS will start it again and again and is always pausing the process repeatedly. It needs to do it to provide run possibilities to all processes. That way, we experience all processes running at the same time. In reality, processes do not run at the same time on a single processor, but they get time slots to run often. If you have more than one CPU in the machine, then processes can actually run at the same time, as many CPUs as there are. As the integration gets more advanced today, desktop computers have CPUs that contain multiple cores that function almost like separate CPUs. On my machine, I have four cores, each capable of executing two threads simultaneously; so, my Mac is almost like an 8 CPU machine. Processes have separate memories. They are allowed to use one part of the memory and if a process tries to use another part that does not belong to it, the processor will stop doing so. The OS will kill the process. Just imagine how frustrated the developers of the original UNIX could have been that they named the program to stop a process to kill, and stopping a process is called killing it. It is like medieval ages when they cut off the hand of a felon. You touch the wrong part of the memory and get killed. I would not like to be a process. The memory handling by the operating system is very complex in addition to separating the processes from each other. When there is not enough memory, the OS writes part of the memory to disk freeing up the memory and reloading that part when it is needed again. This is a very complex, low-level implemented and highly optimized algorithm that is the responsibility of the OS. Threads When I said that the OS executes the processes in time slots, I was not absolutely precise. Every process has one or more threads, and threads are executed. A thread is the smallest execution managed by an external scheduler. Older operating systems did not have the notion of a thread and were executing processes. As a matter of fact, the first thread implementations were simply duplications of processes that were sharing the memory. You may hear the terminology, lightweight process—it means a thread. The important thing is that the threads do not have their own memory. They use the memory of the process. In other words, the threads that run in the same process have undistinguished access to the same memory segment. It is an extremely powerful possibility to implement parallel algorithms that make use of the multiple cores in the machine, but at the same time, it may lead to bugs. Imagine that two threads increment the same long variable. The increment first calculates the incremented value of the lower 32 bits and then the upper, if there were any overflow bits. These are two or more steps that may be interrupted by the OS. It may happen that one thread increments the lower 32 bits, remembers that there is something to do to the upper 32 bits, starts the calculation, but has no time to store the result before it gets interrupted. Then, another thread increments the lower 32 bits, the upper 32 bits, and then the first thread just saves the upper 32 bits that it calculated. The result gets garbled. On an older 32-bit Java implementation, it was extremely easy to demonstrate this effect. On a 64-bit Java implementation, all the 64 bits are loaded into registers and saved back to the memory in one step so it is not that easy to demonstrate multithread issues, but it does not mean that there are none. When a thread is paused and another thread is started, the operating system has to perform a context switch. It means that, among other things, the CPU registers have to be saved and then set to the value that they should have for the other thread. A context switch is always saving the state of the thread and loading the previously saved state of the thread to be started. This is on a CPU register level. This context switch is time consuming; therefore, the more context switches that are done, the more CPU resource is used for the thread administration instead of letting them run. On the other hand, if there are not enough switches, some threads may not get enough time slots to execute, and the program hangs. Fibers Java does not have fibers, but as there are some libraries that support fiber handlings, it is worth mentioning. A fiber is a finer unit than a thread. A program code executing in a thread may decide to give up the execution and tell the fiber manager to just execute some other fiber. What is the point and why is it better than using another thread? The reason is that this way, fibers can avoid part of the context switch. A context switch cannot be avoided totally because a different part of the code that starts to execute it may use the CPU registers in a totally different way. As it is the same thread, the context switching is not the task of the OS, but the application. The OS does not know if the value of a register is used or not. There are bits in the registers, and no one can tell seeing only the processor state whether those bits are relevant for the current code execution or just happen to be there in that way. The program generated by a compiler does know which registers are important and which are those that can just be ignored. This information changes from place to place in the code, but when there is a need for a switch, the fiber passes the information of what is needed to be switched at that point to the code that does the switching. The compiler calculates this information, but Java does not support fibers in the current version. The tools that implement fibers in Java analyze and modify the byte code of the classes to do this after the compilation phase. Golang's goroutines are fibers and that is why you can easily start many thousand goroutines in Go, but you better limit the number of threads in Java to a lower number. They are not the same things. As the terminology lightweight process is fading out and used by less and less fibers, many times are referred to as lightweight threads. java.lang.Thread As everything in Java (well, almost) is object, if we want to start a new thread, we will need a class that represents the thread. This class is java.lang.Thread built into the JDK. When you start a Java code, the JVM automatically creates a few Thread objects and uses them to run different tasks that are needed by it. If you start up VisualVM, you can select the Threads tab of any JVM process and see the actual threads that are in the JVM. For example, the VisualVM as I started it has 29 live threads. One of them is the thread named main. This is the one that starts to execute the main method (surprise!). The main thread started most of the other threads. When we want to write a multithread application, we will have to create new Thread objects and start them. The simplest way to do that is new Thread(), and then calling the start method on the thread. It will start a new Thread that will just finish immediately as we did not give it anything to do. The Thread class, as it is in the JDK, does not do our business logic. The following are the two ways to specify the business logic: Creating a class that implements the Runnable interface Creating a class that extends the Thread class and overrides the run method The following block of code is a very simple demonstration program: public class ThreadIntermingling { static class MyThread extends Thread { private final String name; MyThread(String name){ this.name = name; } @Override public void run(){ for(int i = 1 ; i < 1000 ; i ++ ){ System.out.print(name + " " + i+ ", "); } } } public static void main(String[] args){ Thread t1 = new MyThread("t1"); Thread t2 = new MyThread("t2"); t1.start(); t2.start(); System.out.print("started "); } } The preceding code creates two threads and starts them one after the other. When the start method is called, it schedules the thread object to be executed and then returns. As a result, the new thread will soon start executing asynchronously while the calling thread continues its execution. The two threads, and the main thread, run parallel in the following example and create an output that looks something like this: started t2 1, t2 2, t2 3, t2 4, t2 5, t2 6, t2 7, t2 8, t1 1, t2 9, t2 10, t2 11, t2 12,... The actual output changes from run to run. There is no definite order of the execution or how the threads get access to the single screen output. There is not even guarantee that in each and every execution, the message started is printed before any of the thread messages. To get a better understanding of this, we will have to look at the state diagram of threads. A Java Thread can be in one of the following states: NEW RUNNABLE BLOCKED WAITING TIMED_WAITING TERMINATED These states are defined in the enumThread.State. When you create a new thread object, it is in the NEW state. At this moment, the thread is nothing special, it is just an object but the operating system executionscheduling does not know about it. In some sense, it is only a piece of memory allocated by the JVM. When the start method is invoked, the information about the thread is passed to the operating system and the OS schedules the thread so it can be executed by it when there is an appropriate time slot. Doing this is a resourceful action and that is the reason why we do not create and, especially, do not start new Thread objects only when it is needed. Instead of creating new Threads, we will keep the existing threads for a while, even if they are not needed at the moment, and reuse an existing one if there is one suitable. A thread in the OS can also be in a running state as well as runnable when the OS schedules and executes it at the moment. Java JDK API does not distinguish between the two for good reason. It would be useless. When a thread is in the RUNNABLE state asking if it is actually running from the thread itself, it will result in an obvious answer: if the code just returned from the getState method implemented in the Thread class, then it runs. If it were not running, it would not have returned from the call in the first place. If the getState method was called from another thread, then the result about the other thread by the time the method returns would be meaningless. The OS may have stopped, or started, the queried thread several times until then. A thread is in a BLOCKED state when the code executing in the thread tries to access some resource that is not currently available. To avoid constant polling of resources, the operating system provides effective notification mechanism so the threads get back to the RUNNABLE state when the resource they need becomes available. A thread is in the WAIT or TIMED_WAITING state when it waits for some other thread or lock. TIMED_WAITING is the state when the waiting started calling a version of a method that has timeout. Finally, the TERMINATED state is reached when the thread finishes its execution. If you append the following lines to the end of our previous example, then you will get a TERMINATED printout and also an exception thrown up to the screen complaining about illegal thread state, which is because you cannot start an already terminated thread: System.out.println(); System.out.println(t1.getState()); System.out.println(); t1.start(); Instead of extending the Thread class to define what to execute asynchronously, we can create a class that implements Runnable. Doing that is more in line with the OO programming approach. The something that we implement in the class is not a functionality of a thread. It is more of a something that can be executed. It is something that can just run. If this execution is asynchronous in a different thread, or it is executed in the same thread that was calling the run method, is a different concern that has to be separated. If we do it that way, we can pass the class to a Thread object as a constructor argument. Calling start on the Thread object will start the run method of the object we passed. This is not the gain. The gain is that we can also pass the Runnable object to an Executor (dreadful name, huhh!). Executor is an interface, and implementations execute Runnable (and also Callable, see later) objects in Threads in an efficient way. Executors usually have a pool of Thread objects that are prepared, and in the BLOCKED state. When the Executor has a new task to execute, it gives it to one of the Thread objects and releases the lock that is blocking the thread. The Thread gets into the RUNNABLE state, executes the Runnable, and gets blocked again. It does not terminate and thus can be reused to execute another Runnable later. That way, Executors avoid the resource consuming process of thread registration into the operating system. Professional application code never creates a new Thread. Application code uses some framework to handle the parallel execution of the code or uses Executors provided by some ExecutorService to start Runnable or Callable objects. Pitfalls We have already discussed many of the problems that we may face when developing parallel program. In this section, we will summarize them with the usual terminology used for the problems. Terminology is not only interesting, but it is also important when you talk with colleagues to easily understand each other. Deadlocks Deadlock is the most infamous parallel programming pitfall, and for this reason, we will start with this one. To describe the situation, we will follow the metaphor of bureaucrats. The bureaucrat has to stamp a paper he has in his hand. To do that, he needs the stamp, and he also needs the inkpad. First, he goes to the drawer where the stamp is and takes it. Then, he walks to the drawer where the inkpad is and takes the inkpad. He inks the stamp, pushes on the paper. Then, he puts the stamp back to its place and then the inkpad back in its place. Everything is nice, we are on cloud 9. What happens if another bureaucrat takes the inkpad first and then the stamp second? They may soon end up as one bureaucrat with the stamp in hand waiting for the inkpad and another one with the inkpad in hand waiting for the stamps. And, they may just stay there, frozen forever, and then more and more start to wait for these locks, the papers never get stamped, and the whole system sinks into anarchy. To avoid such situations, the locks have to be ordered and the locks should always be acquired in the order. In the preceding example, the simple agreement that the inkpad is acquired first and the stamp second solves the problem. Whoever acquired the stamp can be sure that the inkpad is free or will soon be free. Race conditions We talk about race conditions when the result of a calculation may be different based on the speed and CPU access of the different parallel running threads. Let's take a look at the following two code lines: void method1(){ 1 a = b; 2 b = a+1; } void method2(){ 3 c = b; 4 b = c+2; } If the value of b at the start of the execution is 0, and two different threads execute the two methods, then the order of the lines can be 1234, 1324, 1342, 3412, 3142, or 3142. Any execution order of the four lines may happen which assures that 1 runs before 2 and 3 runs before 4, but no other restrictions. The outcome, the value of b, is either 1 or 2 at the end of the execution of the segments, which may not be good and what we wanted when coding. Note that the implementation of the parallel Mastermind game also has something like this. The actual guesses very much depend on the speed of the different threads, but this is irrelevant from the final result point of view. We may have different guesses in different runs and that way the algorithm is not deterministic, but we are guaranteed to find the final solution. Overused locks In many situations, it may happen that the threads are waiting on a lock, which protects a resource from concurrent access. If the resource cannot be used by multiple threads simultaneously, and there are more threads than can be served, then the threads are starving. However, in many cases, the resource can be organized in a way so that the threads can get access to some of the services that the resource provides, and the locking structure can be less restrictive. In that case, the lock is overused and the situation can be mended without allocating more resource for the threads. It may be possible to use several locks that control the access to the different functionality of the resource. Starving Starving is the situation when several threads are waiting for a resource trying to acquire a lock and some threads get access to the lock only after extremely long time or never. When the lock is released and there are threads waiting for it, then one of the threads can get the lock. There is usually no guarantee that a thread gets the lock if it waits long enough. Such a mechanism would require intensive administration of the threads, sorting them in the waiting queue. As locking should be a low latency and high performance action, even a few CPU clock cycles are significant; therefore, the locks do not provide this type of fair access by default. Not wasting time with fairness in thread scheduling is a good approach, in case the locks have one thread waiting. The main goal of locks is not scheduling the waiting threads, but rather preventing parallel access to resources. It is like in a shop. If there is somebody at the cashier, you wait. It is a lock built in implicitly. It is not a problem if people do not queue up for the cashier, so long as long there is almost always one free. However, when there are several queues built up in front of the cashiers, then having no queue and waiting order will certainly lead to some very long waiting order for someone who is slow to get access to the cashier. Generally, the solution of fairness and creating queue of waiting threads (customers) is not a good solution. The good solution is to eliminate the situation that leads to waiting queues. You can employ more cashiers, or you can do something totally different that makes the peak load smaller. In a shop, you can give discount to drive customers who come in at off-peak hours. In programming, several techniques can be applied, usually, depending on the actual business we code and fair scheduling of locks is usually a workaround. ExecutorService is an interface in the JDK. An implementation of the interface can execute a Runnable or Callable class in an asynchronous way. The interface only defines the API for the implementation and does not require that the invocation is asynchronous but, in reality, that is the main point implementing such a service. Invoking the run method of a Runnable interface in a synchronous way is simply calling a method. We do not need a special class for that. ExecutorService The Runnable interface defines one run method. It has no arguments returns no value and does not throw any exception. The Callable interface is parameterized and the only method it defines, call, has no argument but returns a generic value and may also throw Exception. In our code, we will implement Runnable if we just want to run something, and Callable when we want to return something. Both of these interfaces are functional interfaces, therefore, they are good candidates to be implemented using lambda. To have an instance of an implementation of an ExecutorService, we can use the utility class Executors. Many times when there is an XYZ interface in the JDK, there can be an XYZs (plural) utility class that provides factory for the implementations of the interface. If we want to start the t1 task many times, we can do so without creating a new Thread. We should use the following executor service: public class ThreadIntermingling { static class MyThread implements Runnable { private final String name; MyThread(String name) { this.name = name; } @Override public void run() { for (int i = 1; i < 1000; i++) { System.out.print(name + " " + i + ", "); } } } public static void main(String[] args) throws InterruptedException, ExecutionException { ExecutorService es = Executors.newFixedThreadPool(2); Runnable t1 = new MyThread("t1"); Runnable t2 = new MyThread("t2"); Future<?> f1 = es.submit(t1); Future<?> f2 = es.submit(t2); System.out.print("started "); f1.get(); f2.get(); System.out.println(); f1 = es.submit(t1); es.shutdown(); } } This time, we do not get any exception. Instead, the t1 task runs second time. In this example, we are using a fixed size thread pool that has two Threads. As we want to start only two threads simultaneously, it is enough. There are implementations that grow and shrink the size of the pool dynamically. Fixed size pool should be used when we want to limit the number of the threads or we know from some other information source the number of the a-priory threads. In this case, it is a good experiment to change the size of the pool to one and see that the second task will not start in this case until the first one finishes. The service will not have another thread for t2 and will have to wait until the one and only Thread in the pool is freed. When we submit the task to the service, it returns even if the task cannot currently be executed. The tasks are put in a queue and will start execution as soon as there is enough resource to start them. The submit method returns a Future object, as we can see in the preceding sample. It is like a service ticket. You bring your car to the repair mechanic, and you get a ticket. You are not required to stay there until the car is fixed, but at any time, you can ask if the car is ready. All you need is the ticket. You can also decide to wait until the car is ready. A Future object is also something like that. You do not get the value that you need. It will be calculated asynchronously. However, there is a Future promise that it will be there and your ticket to access the object you need is the Future object. When you have a Future object, you can call the isDone method to see if it is ready. You can start waiting for it to call get with, or without, some timeout. You can also cancel the task executing it, but in that case, the outcome may be questionable. Just like, in case of your car, if you decide to cancel the task, you may get back your car with the motor disassembled. Similarly, cancelling a task that is not prepared for it may lead to resource loss, opened and inaccessible database connection (this is a painful memory for me, even after 10 years), or just a garbled unusable object. Prepare your tasks to be cancelled or do not cancel them. In the preceding example, there is no return value for Future because we submitted a Runnable object and not a Callable one. In that case the value passed to the Future is not to be used. It is usually null, but that is nothing to lean on. The final and most important thing that many developers miss, even me, after not writing multithread Java API using code for years, is shutting down the ExecutorService. The ExecutorService is created and it has Thread elements. The JVM stops when all non-daemon threads are stopped. It ain't over till the fat lady sings. A thread is a daemon thread if it was set to be daemon (invoking setDaemon(true)) before it was started. A thread is automatically daemon of the starting thread is a daemon thread. Daemon threads are stopped by the JVM when all other threads are finished and the JVM wants to finish. Some of the threads the JVM executes itself are daemon threads, but it is likely that there is no practical use of creating daemon threads in an application program. Not shutting down the service simply prevents the JVM from stopping. The code will hang after the main method finishes. To tell the ExecutorService that there is no need for the threads it has, we will have to shutdown the service. The call will only start the shutdown and return immediately. In this case, we do not want to wait. The JVM does anyway. If we need to wait, we will have to call awaitTermination. ForkJoinPool The ForkJoinPool is a special ExecutorService that has methods to execute ForkJoinTask objects. These classes are very handy when the task that we want to perform can be split into many small tasks and then the results, when they are available, aggregated. Using this executor, we need not care about the size of the thread pool and shutting down the executor. The size of the thread pool is adjusted to the number of processors on the given machine to have optimal performance. As the ForkJoinPool is a special ExecutorService that is designed for short running tasks, it does not expect any task to be there longer or being needed when there are no more tasks to run. Therefore, it is executed as a daemon thread; when the JVM shuts down, the ForkJoinPool automatically stops and the lady does not sing any more. To create a task, the programmer should extend either RecursiveTask or RecursiveAction. The first one is to be used when there is some return value from the task, the second when there is no computed value returned. They are called recursive because many times, these tasks split the problem they have to solve smaller problems and invoke these tasks asynchronously through the fork-join API. A typical problem to be solved using this API is the quick-sort. In the Chapter 3, Optimizing the Sort Making Code Professional we created two versions of the quick-sort algorithm. One using recursive calls and one without using it. We can also create a new one, which, instead of calling itself recursively, schedule the task to be executed, perhaps by another processor. The scheduling is the task of the ForkJoinPool implementation of ExecutorService. You may revisit the code of Qsort.java in Chapter 3, Optimizing the Sort - Making Code Professional. Here is the version that is using ForkJoinPool: public class FJQuickSort<E> { final private Comparator<E> comparator; final private Swapper swapper; public FJQuickSort(Comparator<E> comparator, Swapper swapper){ this.comparator = comparator; this.swapper = swapper; } public void qsort(SortableCollection<E> sortable, int start, int end) { ForkJoinPool pool = new ForkJoinPool(); pool.invoke(new RASort(sortable,start,end)); } private class RASort extends RecursiveAction { final SortableCollection<E> sortable; final int start, end; public RASort(SortableCollection<E> sortable, int start, int end) { this.sortable = sortable; this.start = start; this.end = end; } public void compute() { if (start < end) { final E pivot = sortable.get(start); final Partitioner<E> partitioner = new Partitioner<>(comparator, swapper); int cutIndex = partitioner.partition( sortable, start, end, pivot); if (cutIndex == start) { cutIndex++; } RecursiveAction left = new RASort(sortable, start, cutIndex - 1); RecursiveAction right = new RASort(sortable, cutIndex, end); invokeAll(left,right); left.join(); right.join(); } } } Whenever you can split your tasks into subtasks similar to the way it was done in the preceding quick-sort example, I recommend that you use ForkJoinPool as an ExecutorService. You can find good documentation on the API and the use on the JavaDoc documentation of Oracle. Variable access Now that we can start threads and create code that runs parallel, it is time to talk a little bit about how these threads can exchange data between each other. At first glimpse, it seems fairly simple. The threads use the same shared memory; therefore, they all can read and write all the variables that the Java access protection allows them. This is true, except that some threads may just decide not to read the memory. After all, if they have just recently read the value of some variable, why read it again from the memory to the registers if it was not modified? Who would have modified them? Let's see the following short example: package packt.java9.by.example.thread; public class VolatileDemonstration implements Runnable { private Object o = null; private static final Object NON_NULL = new Object(); @Override public void run() { while( o == null ); System.out.println("o is not null"); } public static void main(String[] args) throws InterruptedException { VolatileDemonstration me = new VolatileDemonstration(); new Thread(me).start(); Thread.sleep(1000); me.o = NON_NULL; } } What will happen? You may expect that the code starts up, starts the new thread, and one minute, when the main thread sets the object to something not null, will it stop? It will not. It may stop on some Java implementations, but in most of them, it will just keep spinning. The reason for that is that the JIT compiler optimizes the code. It sees that the loop does nothing and also that the variable will just never be non-null. It is allowed to assume that because the variables not declared volatile are not supposed to be modified by any other thread, the JIT is eligible to optimize. If we declare the Object o variable to be volatile (with the volatile keyword), then the code will stop. In case you try to remove the call to sleep, the code will also stop. This, however, does not fix the issue. The reason is that JIT optimization kicks in only after about 5000 loops of the code execution. Before that, the code runs naive and stops before the optimization will eliminate the extra and regularly not needed access to the non-volatile variable. If this is so gruesome, then why don't we declare all variables to be volatile? Why does Java not do that for us? The answer is speed, and to understand it deeper, we will use our metaphor, the office, and the bureaucrat. The CPU heartbeat These days CPUs run on 2 to 4 GHz frequency processors. It means that a processor gets 2 to 4 times 109 clock signals to do something every second. A processor cannot do any atomic operation faster than this, and also there is no reason to create a clock that is faster than what a processor can follow. It means that a CPU performs a simple operation, such as incrementing a register in half or quarter of a nanosecond. This is the heartbeat of the processor, and if we think of the bureaucrat as humans, who they are, then it is equivalent to one second, approximately, if and as their heartbeat. Processors have registers and caches on the chip on different levels, L1, L2, and sometimes L3; there is memory, SSD, disk, network, and tapes that may be needed to retrieve data. Accessing data that is in the L1 cache is approximately 0.5ns. You can grab a paper that is on your desk— half of a second. L2 cache is 7ns. This is a paper in the drawer. You have to push the chair a bit back, bend it in a sitting position, pull out the drawer, take the paper, push the drawer back, and raise and put the paper on the desk; it takes 10 seconds, give or take. Main memory read is 100ns. The bureaucrat stands up, goes to the shared file at the wall, he waits while other bureaucrats are pulling their papers or putting theirs back, selects the drawer, pulls it out, takes the paper, and walks back to the desk. This is two minutes. This is volatile variable access every time you write a single word on a document and it has to be done twice. Once to read, and once to write, even if you happen to know that the next thing you will do is just fill another field of the form on the same paper. Modern architectures, where there are no multiple CPUs but rather single CPUs with multiple cores, are a bit faster. One core may check the other core's caches to see if there was any modification on the same variable, but this speeds the volatile access to 20ns or so, which is still a magnitude slower than nonvolatile. Although the rest is less focused on multithread programming, it is worth mentioning here, because it gives good understanding on the different time magnitudes. Reading a block from an SSD (4K block usually) is 150,000ns. In human speed, that is a little bit more than 5 days. Reading or sending something to a server over the network on the Gb local Ethernet is 0.5ms, which is like waiting for almost a month for the metaphoric bureaucrat. If the data over the network is on a spinning magnetic disk, then seek time adds up (the time until the disk rotates so that the part of the magnetic surface gets under the reading head) to 20ms. It is, approximately, a year in human terms. If we send a network packet over the Atlantic on the Internet, it is approximately is 150ms. It is like 14 years, and this was only one single package; if we want to send data over the ocean, it may be seconds that count up to historic times, thousands of years. If we count one minute for a machine to boot, it is equivalent to the time span of our whole civilization. We should consider these numbers when we want to understand what the CPU is doing most of the time: it waits. Additionally, it also helps cool your nerves when you think about the speed of a real-life bureaucrat. They are not that slow after all, if we consider their heartbeat, which implies the assumption that they have a heart. However, let's go back to real life, CPUs, and L1, L2 caches and volatile variables. Volatile variables Let's modify the declaration of the o variable in our sample code as follows: private volatile Object o = null; The preceding code runs fine and stops after a second or so. Any Java implementation has to guarantee that multiple threads can access volatile fields and the value of the field is consistently updated. This does not mean that volatile declaration will solve all synchronization issues, but guarantees that the different variables and their value change relations are consistent. For example, let's consider we have the following two fields incremented in a method: private volatile int i=0,j=0; public void method(){ i++; j++; } In the preceding code, reading i and j from another thread will never result an i>j. Without the volatile declaration, the compiler is free to reorganize the execution of the increment operations if it needs and thus, it will not guarantee that an asynchronous thread reads consistent values. Synchronized block Declaring variables are not the only tool to ensure the consistency between threads. There are other tools in the Java language and one of them is the synchronized block. The synchronized keyword is part of the language and it can be used in front of a method or a program block inside a method. Every object in the Java program has a monitor that can be locked and unlocked by any running thread. When a thread locks a monitor, it is said that that thread holds the lock, and no two threads can hold the lock of a monitor at a time. If a thread tries to lock a monitor that is already locked, it gets BLOCKED until the monitor is released. A synchronized block starts with the synchronized keyword, and then an object instance specified between parentheses and the block comes. The following small program demonstrates the synchronized block: public class SynchronizedDemo implements Runnable { public static final int N = 1000; public static final int MAX_TRY = 1_000_000; private final char threadChar; private final StringBuffer sb; public SynchronizedDemo(char threadChar, StringBuffer sb) { this.threadChar = threadChar; this.sb = sb; } @Override public void run() { for (int i = 0; i < N; i++) { synchronized (sb) { sb.append(threadChar); sleep(); sb.append(threadChar); } } } private void sleep() { try { Thread.sleep(1); } catch (InterruptedException ignored) {} } public static void main(String[] args) { boolean failed = false; int tries = 0; while (!failed && tries < MAX_TRY) { tries++; StringBuffer sb = new StringBuffer(4 * N); new Thread(new SynchronizedDemo('a', sb)).start(); new Thread(new SynchronizedDemo('b', sb)).start(); failed = sb.indexOf("aba") != -1 || sb.indexOf("bab") != -1; } System.out.println(failed ? "failed after " + tries + " tries" : "not failed"); } } The code starts two different threads. One of the threads appends aa to the StringBuffer. The other one appends bb. This appending is done in two separate steps with a sleep in between. The sleep is needed to avoid JIT that optimizes the two separate steps into one. Each thread executes the append 1000 times each time appending a or b two times. As the two appends one after the other are inside a synchronized block it cannot happen that an aba or bab sequence gets into the StringBuffer. While one thread executes the synchronized block, the other thread cannot execute it. If I remove the synchronized block, then the JVM I used to test Java HotSpot (TM) 64-Bit Server VM (build 9-ea+121, mixed mode) prints out the failure with a try-count around a few hundreds. It clearly demonstrates what the synchronization means, but it draws our attention to another important phenomena. The error occurs only around every few hundred thousand executions only. It is extremely rare, even though this example was furnished to demonstrate such a mishap. If a bug appears so rare, it is extremely hard to reproduce and, even more, to debug and fix. Most of the synchronization errors manifest in mysterious ways and their fixing usually is the result of meticulous code review rather than debugging. Therefore, it is extremely important to clearly understand the true nature of Java multithread behavior before starting commercial multithread application. The synchronized keyword can also be used in front of a method. In this case, the object to acquire the lock of is the object. In case of a static method, the synchronization is done on the whole class. Wait and notify There are five methods implemented in the class Object that can be used to get further synchronization functionality: wait with three different timeout argument signature, notify, and notifyAll. To call wait, the calling thread should have the lock of the Object on which wait is invoked. It means that you can only invoke wait from inside a synchronized block, and when it is called, the thread gets BLOCKED and releases the lock. When another thread calls notify all on the same Object, the thread gets into the RUNNABLE state. It cannot continue execution immediately as it cannot get the lock on the object. The lock is held at that moment by the thread that just called notifyAll. However, sometime after the other thread releases, the lock gets out of the synchronized block, and the waiting thread continues the execution. If there are more threads waiting on an object, all of them get out of the BLOCKED state. The notify method wakes only one of the waiting threads. There is no guarantee which thread is awakened. The typical use of wait, notify, and notifyAll is when one or more threads are creating Objects that are consumed by other thread, or threads. The storage where the objects travel between the threads is some kind of queue. The consumer waits until there is something to read from the queue, and the producer puts the objects into the queue one after the other. The producer notifies the consumers when it stores something into the queue. If there is no room left in the queue, the producer has to stop and wait until the queue has some space. In this case, the producer calls the wait method. To wake the producer up, the consumer calls notifyAll when it reads something. The consumer consumes the objects from the queue in a loop and calls wait only if there is nothing to be read from the queue. When the producer calls notifyAll, and there is no consumer waiting, the notification is just ignored. It flies away, but this is not a problem; consumers are not waiting. When the consumer consumes an object and calls notifyAll and there is no producer waiting, the situation is the same. It is not a problem. It cannot happen that the consumer consumes, calls notifyAll, and after the notification was flying in the air not finding any waiting producer, a producer starts to wait. This cannot happen because the whole code is in a synchronized block and it ensures that no producer is in the critical section. This is the reason why wait, notify, and notifyAll can only be invoked when the lock of the Object class is acquired. If there are many consumers, which are executing the same code and are equivalently good in consuming the objects, then it is an optimization to call notify instead of notifyAll. In that case, notifyAll will just awake all consumer threads and all, but the lucky one will recognize that they were woken up but somebody else already got away with the bait. I recommend that you practice at least once to implement a blocking queue that can be used to pass Objects between threads. However, never use that code in production: starting with Java 1.5, there are implementations of the BlockingQueue interface. Use one that fits your needs. We will too, in our example code. Feel lucky that you can code in Java 9. I started using Java professionally when it was 1.4 and once I had to implement a blocking queue. Life gets just better and easier all the time with Java. In professional code, we usually avoid using synchronized methods or blocks and volatile fields as well as the wait and notify methods, notifyAll too, if possible. We can use asynchronous communication between threads, or pass the whole multithreading to the framework for handling. Synchronized and volatile cannot be avoided in some special cases when the performance of the code is important, or we cannot find a better construct. Sometimes, the direct synchronization on specific code and data structures is more efficient than the approach delivered by JDK classes. It is to note, however, that those classes also use these lowlevel synchronization constructs, so it is not magic how they work; and to develop yourself, you can look into the code of the JDK classes before you want to implement your own version. You will realize that it is not that simple to implement these queues; the code of the classes is not complex and compound without reason. If you find the code simple, it means that you are senior enough to know what not to reimplement. Or, perhaps, you do not even realize what code you read. Lock s are built in Java; every Object has a lock that a thread may acquire when it enters a synchronized block. We discussed that already. In some programming code, there are situations when this kind of structure is not optimal. Lock In some situations, the structure of locks may be lined up to avoid deadlock. It may be needed to acquire lock A before B and to acquire B before C. However, A should be released as soon as possible, not to prevent access to resource protected by lock D, but also needing lock A before it. In complex and highly parallel structures, the locks are structured many times into trees where accessing a resource a thread should climb down along the tree to a leaf representing the resource. In this climbing, the thread gets hold of a lock on a node, then a lock on a node below it, and then releases the lock above, just like a real climber descending (or climbing up if you imagine the tree with the leafs at the top, which is more realistic, nevertheless graphs usually show trees upside down). You cannot leave a synchronized block remaining in another that is inside the first one. Synchronized blocks are nested. The java.util.concurrent.Lock interface defines methods to handle that situation and the implementations are also there in the JDK to be used in our code. When you have a lock, you can call the methods lock and unlock. The actual order is in your hand and you can write the following line of code to get the locking sequence: a.lock(); b.lock(); a.unlock(); c.lock() The freedom, however, comes with responsibility as well. The locks and unlocks are not tied to the execution sequence of the code, like in case of synchronized block, and it may be very easy to create code that in some case just loses a lock not unlocking it rendering some resource unusable. The situation is similar to a memory leak: you will allocate (lock) something and forget to release (unlock) it. After a while, the program will run out of resource. My personal recommendation is to avoid using locks if possible and use higher-level constructs and asynchronous communications between threads, such as blocking queues. Condition The java.util.concurrent.Condition interface in functionality is similar to the built-in wait, notify, and notifyAll. Any implementation of Lock should create new Condition objects and return as a result to the invocation of the newCondition method. When the thread has a Condition, it can call await, signal, and signalAll when the thread has the lock that created the condition object. The functionality is very similar to the methods of Object mentioned. However, the big difference is that you can create many Condition for a single Lock and they will work independent of each other, but not independent of the Lock. ReentrantLock is the simplest implementation of the interface lock in the JDK. There are two ways to create this type of lock: with and without fairness policy. If the ReentrantLock(Boolean fair) constructor is called with the true argument, then the lock will be assigned to the thread that is waiting for the lock the longest time in case there are many threads waiting. This will avoid a thread made to wait for infinite time and starving. ReentrantLock ReentrantReadWriteLock This class is an implementation of ReadWriteLock. ReadWriteLock is a lock that can be used for parallel read access and exclusive write access. It means that several threads can read the resource protected by the lock, but when a thread writes the resource, no other thread can get access to it, not even read during that period. A ReadWriteLock is simply two Lock objects returned by the readLock and writeLock methods. To get read access on ReadWriteLock, the code has to invoke myLock.readLock().lock(), and to get access to write lock, myLock.writeLock().lock(). Acquiring one of the locks and releasing it in the implementation is coupled with the other lock. To acquire a write lock, no thread should have an active read lock, for example. There are several intricacies in the use of the different lock. For example, you can acquire a read lock, but you cannot get a write lock so long as you have the read lock. You have to release the read lock first to acquire a write lock. This is just one of the simple details, but this is the one that novice programmers have trouble with many times. Why is it implemented this way? Why should the program get a write lock, which is more expensive—in sense of higher probability locking other threads—when it still is not sure that it wants to write the resource? The code wants to read it and. based on the content. it may later decide that it wants to write it. The issue is not with the implementation. The developers of the library decided this rule, not because they just liked it that way or because they were aware of parallel algorithms and deadlock possibilities. When two threads have read lock and each decides to upgrade the lock to write lock, then they would intrinsically create a deadlock. Each would hold the read lock waiting for the write and none of them would get it ever. On the other end, you can downgrade a write lock to a read lock without risking that in the meantime somebody acquires a write lock and modifies the resource. Atomic classes Atomic classes enclose primitive values into objects and provide atomic operations on them. We discussed race conditions and volatile variables. For example, if we have an int variable to be used as a counter and we want to assign a unique value to objects that we work with, we can increment the value and use the result as a unique ID. However, when multiple threads use the same code, we cannot be sure about the value we read after the increment. It may happen that another thread also incremented the value in the meantime. To avoid that, we will have to enclose the increment and the assignment of the incremented value to an object into a synchronized block. This can also be done using AtomicInteger. If we have a variable of AtomicInteger, then calling incrementAndGet increments the value of int enclosed in the class and returns the incremented value. Why do it instead of using synchronized block? The first answer is that if the functionality is there in the JDK, then using it is less line than implementing it again. Developers maintaining the code you create are expected to know the JDK libraries but have to study your code, and this takes time and time is money. The other reason is that these classes are highly optimized and, many times, they implement the features using platform specific native code that greatly over performs the version we can implement using synchronized blocks. Worrying about performance too early is not good, but parallel algorithms and synchronization between threads are usually used when performance is crucial; thus, there is a good chance that the performance of the code using the atomic classes is important. In the java.util.concurrent.atomic package, there are several classes, AtomicInteger, AtomicBoolean, AtomicLong, and AtomicReference among them. They all provide methods that are specific to the encapsulated value. The method, which is implemented by every atomic class, is compareAndSet. This is a conditional valuesetting operation that has the following format: boolean compareAndSet(expectedValue, updateValue); When it is applied on an atomic class, it compares the actual value with the one expectedValue, and if they are the same, then it sets the value to updateValue. If the value was updated, the method returns true and it does all this in an atomic action. You may ask the question that if this method is in all of these classes, why is there no Interface defining this method? The reason for this is that the argument types are different based on the encapsulated type, and these types are primitives. As primitives cannot be used as generic types, not even a generic interface can be defined. In case of AtomicXXXArray, the method has an extra first argument, which is the index of the array element handled in the call. The variables encapsulated are handled the same way as volatile, as far as the reordering is concerned, but there are special methods that loosen the conditions a bit to be used when possible, and performance is key. The general advice is to consider using atomic classes, if there is one usable, and you will find yourself creating a synchronized block for check-and-set, atomic increment, or addition operations. BlockingQueue is an interface that extends the standard Queue interface with methods that are suitable to be used by multithread applications. Any implementation of this interface provides methods that allow different threads to put element into the queue, pull elements off the queue, and wait for elements that are in the queue. BlockingQueue When there is a new element that is to be stored in the queue, you can add it, offer it, or put it. These are the name of the methods that store elements and they do the same thing, but a bit differently. The add element throws an exception if the queue is full and there is no room for the element. The offer element does not throw exception but returns either true or false, based on the success. If it can store the element in the queue, it returns true. There is also a version of offer that specifies a timeout. That version of the method waits, and returns only false if it cannot store the value into the queue during the period. The put element is the dumbest version; it waits until it can do its job. When talking about available room in a queue, do not get puzzled and mix it with general Java memory management. If there is no more memory, and the garbage collector can also not release any, you will certainly get OutOfMemoryError. Exception is thrown by add, and false is returned by offer, when the queue limits are reached. Some of the BlockingQueue implementations can limit the number of elements that can be stored at a time in a queue. If that limit is reached, then the queue is full and cannot accept more elements. Fetching elements from a BlockingQueue implementation also has four different ways. In this direction, the special case is when the queue is empty. In that case, remove throws an exception instead of returning the element, poll returns null if there is no element, and take just waits until it can return an element. Finally, there are two methods inherited from the interface Queues that do not consume the element from the queue only look at. The element return the head of the queue and throws an exception if the queue is empty, and peek returns null if there is no element in the queue. The following table summarizes the operations borrowed from the documentation of the interface: Throws exception Special value Blocks Times out Insert add(e) offer(e) put(e) offer(e, time, unit) Remove remove() poll() take() poll(time, unit) Examine element() peek() not applicable not applicable LinkedBlockingQueue This is an implementation of the BlockingQueue interface, which is backed up by a linked list. The size of the queue is not limited by default (to be precise, it is Integer.MAX_VALUE) but it can optionally be limited in a constructor argument. The reason to limit the size in this implementation is to aid the use when the parallel algorithm performs better with limited size queue, but the implementation does not have any restriction on the size. LinkedBlockingDeque This is the simplest implementation of the BlockingQueue and also its subinterface BlockingDeque. As we discussed in the previous chapter, a Deque is a double-ended queue that has add, remove, offer, and so on, type of methods in the form of xxxFirst and xxxLast to do the act with one or the other end of the queue. The Deque interface defines getFirst and getLast instead of consistently naming elementFirst and elementLast, so this is something you should get used to. After all, the IDEs help with automatic code completion so this should not be a really big problem. ArrayBlockingQueue implements the BlockingQueue interface, hence the Queue interface. This implementation manages a queue with fixed size elements. The storage in the implementation is an array and the elements are handled in a FIFO manner: first-in first-out. This is the class that we will also use in the parallel implementation of Mastermind for the communication between the boss and the subordinated bureaucrats. ArrayBlockingQueue LinkedTransferQueue The TransferQueue interface is extending BlockingQueue and the only implementation of it in the JDK is LinkedTransferQueue. A TransferQueue comes handy when a thread wants to hand over some data to another thread and needs to be sure that some other thread takes the element. This TransferQueue has a method transfer that puts an element on the queue but does not return until some other thread removes (or polls) it. That way the producing thread can be sure that the object put on the queue is in the hands of another processing thread and does not wait in the queue. The method transfer also has a format tryTransfer in which you can specify some timeout value. If the method times out the element is not put into the queue. IntervalGuesser We discussed the different Java language elements and JDK classes that are all available to implement parallel algorithms. Now, we will see how to use these approaches to implement the parallel guesser for the Masterrmind game. The class that performs the creation of the guesses is named IntervalGuesser. It creates the guesses between a start and an end guess and sends them to a BlockingQueue. The class implements Runnable so it can run in a separate Thread. The purist implementation will separate the Runnable functionality from the interval guessing, but as the whole class is hardly more than 50 lines, it is forgivable sin implementing the two functionalities in a single class. public class private private private private IntervalGuesser extends UniqueGuesser implements Runnable { final Guess start; final Guess end; Guess lastGuess; final BlockingQueue<Guess> guessQueue; public IntervalGuesser(Table table, Guess start, Guess end, BlockingQueue<Guess> guessQueue) { super(table); this.start = start; this.end = end; this.lastGuess = start; this.guessQueue = guessQueue; nextGuess = start; } @Override public void run() { Guess guess = guess(); try { while (guess != Guess.none) { guessQueue.put(guess); guess = guess(); } } catch (InterruptedException ignored) { } } @Override protected Guess nextGuess() { Guess guess; guess = super.nextGuess(); if (guess.equals(end)) { guess = Guess.none; } lastGuess = guess; return guess; } public String toString() { return "[" + start + "," + end + "]"; } } The implementation is very simple as most of the functionality is already implemented in the abstract Guesser class. The more interesting code is the one that invoked the IntervalGuesser. ParallelGamePlayer The ParallelGamePlayer class implements the Player interface that defines the play method: @Override public void play() { Table table = new Table(NR_COLUMNS, manager); Secret secret = new RandomSecret(manager); Guess secretGuess = secret.createSecret(NR_COLUMNS); Game game = new Game(table, secretGuess); final IntervalGuesser[] guessers = createGuessers(table); startAsynchronousGuessers(guessers); final Guesser finalCheckGuesser = new UniqueGuesser(table); try { while (!game.isFinished()) { final Guess guess = guessQueue.take(); if (finalCheckGuesser.guessMatch(guess)) { game.addNewGuess(guess); } } } catch (InterruptedException ie) { } finally { stopAsynchronousGuessers(guessers); } } This method creates a Table, a RandomSecret that creates the guess used as a secret in a random way, a Game object, IntervalGuessers, and a UniqueGuesser. The IntervalGuessers are the bureaucrats; the UniqueGuesser is the boss who crosschecks the guesses that the IntervalGuessers create. The method starts off the asynchronous guessers and then reads the guesses in a loop from them and puts them on the table if they are OK until the game finishes. At the end of the method, in the finally block, the asynchronous guessers are stopped. The start and the stop method for the asynchronous guessers use ExecutorService. private ExecutorService executorService; private void startAsynchronousGuessers( IntervalGuesser[] guessers) { executorService = Executors.newFixedThreadPool(nrThreads); for (IntervalGuesser guesser : guessers) { executorService.execute(guesser); } } private void stopAsynchronousGuessers( IntervalGuesser[] guessers) { executorService.shutdown(); guessQueue.drainTo(new LinkedList<>()); } The code is quite straightforward. The only thing that may need mention is that the queue of the guesses is drained into a collection that we do not use afterward. This is needed to help any IntervalGuesser that is waiting with a suggested guess in hand, trying to put it into the queue. When we drain the queue, the guesser thread returns from the method put in the guessQueue.put(guess); line in IntervalGuesser and can catch the interrupt. The rest of the code does not contain anything that would be radically different from what we have already seen and you can find it on GitHub. The last question that we still want to discuss in this chapter is how much speed did we gain making the code parallel? Microbenchmarking Microbenchmarking is measuring the performance of a small code fragment. When we want to optimize our code, we will have to measure it. Without measurement, code optimization is like shooting blindfolded. You will not hit the target, but you likely will shoot somebody else. Shooting is a good metaphor because you should usually not do it, but when you really have to then you have no choice. If there is no performance issue and the software meets the requirements, then any optimization, including speed measurement, is a waste of money. This does not mean that you are encouraged to write slow and sloppy code. When we measure performance, we will compare it against a requirement, and the requirement is usually on the user level. Something like, the response time of the application should be less than 2 seconds. To do such a measurement, we usually create load tests in a test environment and use different profiling tools that tell us what is consuming the most time and where we should optimize. Many times, it is not only Java code, but configuration optimization, using larger database connection pool, more memory, and similar things. Microbenchmarking is a different story. It is about the performance of a small Java code fragment and, as such, closer to the Java programming. It is rarely used, and before starting to do a microbenchmark for real commercial environment, we will have to think twice. Microbenchmark is a luring tool to optimize something small without knowing if it is worth optimizing that code. When we have a huge application that has several modules run on several servers, how can we be sure that improving some special part of the application drastically improves the performance? Will it pay back in increased revenue that generates so much profit that will cover the cost we burned into the performance testing and development? Statistically, almost sure that such an optimization including microbenchmarking will not pay off. Once I was maintaining the code of a senior's colleague. He created a highly optimized code to recognize configuration keywords that were present in a file. He created a program structure that represented a decision tree based on the characters in the key string. If there was a keyword in the configuration file that was misspelled, the code threw an exception at the very first character where it could decide that the keyword could not be correct. To insert a new keyword, it needed to get through the code structure to find the occasion in the code where the new keyword was first different from already existing ones and extend the deeply nested if/else structures. To read the list of the handled keywords was possible from the comments that listed all the keywords that he did not forget to document. The code was working blazingly fast, probably saving a few milliseconds of the servlet application startup time. The application was started up only after system maintenance every few month. You feel the irony, don't you? Seniority is not always the number of years. Lucky ones can save their inner child. So when to use microbenchmarking? I can see two areas: You identified the code segment that eats most of the resources in your application and the improvement can be tested by microbenchmarks You cannot identify the code segment that will eat most of the resources in an application but you suspect it The first is the usual case. The second is when you develop a library, and you just do not know all the applications that will use it. In this case, you will try to optimize the part that you think is the most crucial for most of the imagined, suspected applications. Even in that case, it is better to take some sample applications that are created by users of your library and collect some statistics about the use. Why should we talk about microbenchmarking in details? What are the pitfalls? Benchmarking is an experiment. The first programs I wrote was a TI calculator code and I can just count the number of steps the program made to factor two large (10 digits those days) prime numbers. Even at that time, I was using an old Russian stopwatch to measure the time, being lazy to calculate the number of steps. Experiment and measurement was easier. Today, you cannot calculate the number of steps the CPU makes even if you wanted. There are so many small factors that may change the performance of the applications that are out of control of the programmer, which makes it impossible to calculate the steps. We have the measurement left for us, and we will gain all the problems of measurements. What is the biggest problem? We are interested in something, say X, and we usually cannot measure that. So, we will measure Y instead and hope that the values of Y and X are coupled together. We want to measure the length of the room, but instead we measure the time it takes for the laser beam to travel from one end to the other. In this case, the length X and the time Y are strongly coupled. Many times, X and Y only correlate more or less. Most of the times, when people do measurement, the X and Y values have no relation to each other at all. Still, people put their money and more on decisions backed by such measurements. Microbenchmarking is no different. The first question is how to measure the execution time? Small code runs short times and System.currentTimeMillis() may just return the same value when the measurement starts and when it ends, because we are still in the same millisecond. Even if the execution is 10ms, the error of the measurement is still at least 10% purely because of the quantization of the time as we measure. Luckily, there is System.nanoTime(). But is there? Just because the name says it returns the number of nanoseconds from a specific start time, it does not necessarily mean it really can. It very much depends on the hardware and the implementation of the method in the JDK. It is called nano because this is the precision that we cannot certainly reach. If it was microseconds, then some implementation may be limited by the definition, even if on the specific hardware, there is a more precise clock. However, this is not only the precision of an available hardware clock; it is about the precision of the hardware. Let's remember the heartbeat of the bureaucrats, and the time needed to read something from memory. Calling a method, such as System.nanoTime(), is like asking the bellboy in a hotel to run down from the second floor to the lobby and peek out to look at the clock on the tower on the other side of the road, come back, and tell seconds precision what the time was it when we asked. Nonsense. We should know the precision of the clock on the tower and the speed of the bellboy running from the floor to the lobby and back. This is a bit more than just calling nanoTime. This is what a microbenchmarking harness does for us. The Java Microbenchmarking Harness (JMH) is available for some time as a library. It is developed by Oracle and used to tune the performance of some core JDK classes, and with Java 9, these performance measurements and results become part of the distributed JDK. This is good news for those who develop Java platform for new hardware, but also for developers, because it means that the JMH is and will be supported by Oracle. "JMH is a Java harness to build, run, and analyze nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM." (quote from the official site of JMH, http://openjdk.java.net/projects/cod e-tools/jmh/). You can run jmh as a separate project independent from the actual project you measure, or you can just store the measurement code in a separate directory. The harness will compile against the production class files and will execute the benchmark. The easiest way, as I see, is to use the Gradle plugin to execute JMH. You can store the benchmark code in a directory called jmh (the same level as main and test) and create a main that can start the benchmark. The Gradle build script is extended with the following lines: buildscript { repositories { jcenter() } dependencies { classpath "me.champeau.gradle:jmh-gradle-plugin:0.2.0" } } apply plugin: "me.champeau.gradle.jmh" jmh { jmhVersion = '1.13' includeTests = true } And the microbenchmark class is as follows: public class MicroBenchmark { public static void main(String... args) throws IOException, RunnerException { Options opt = new OptionsBuilder() .include(MicroBenchmark.class.getSimpleName()) .forks(1) .build(); new Runner(opt).run(); } @State(Scope.Benchmark) public static class ThreadsAndQueueSizes { @Param(value = {"1", "4", "8"}) String nrThreads; @Param(value = { "-1","1", "10", "100", "1000000"}) String queueSize; } @Benchmark @Fork(1) public void playParallel(ThreadsAndQueueSizes t3qs) throws InterruptedException { int nrThreads = Integer.valueOf(t3qs.nrThreads); int queueSize = Integer.valueOf(t3qs.queueSize); new ParallelGamePlayer(nrThreads, queueSize).play(); } @Benchmark @Fork(1) public void playSimple(){ new SimpleGamePlayer().play(); } } is created to play the game with -1, 1, 4, and 8 IntervalGuesser threads, and in each case, there is a test running with a queue of length 1, 10, 100, and 1 million. These are 16 test executions. When the number of threads is negative, then the constructor uses LinkedBlockingDeque. There is another separate measurement that measures the nonparallel player. The test was executed with unique guesses and secrets (no color used more than once) and ten colors and six columns. ParallelGamePlayer When the harness starts, it does all the calibrations automatically and runs the tests for many iterations to let the JVM start up. You may recall the code that just never stopped unless we used the volatile modifier in for the variable that was used to signal the code to stop. That happened because the JIT compiler optimized the code. This is done only when the code was already run a few thousand times. The harness makes these executions to warm up the code and ensure that the measurement is done when JVM is at full speed. Running this benchmark takes approximately 15 minutes on my machine. During the execution, it is recommended to stop all other processes and let the benchmark use all available resources. If there is anything using resources during the measurement, then it will be reflected in the result. Benchmark playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playParallel playSimple (nrThreads) 1 1 1 1 1 4 4 4 4 4 8 8 8 8 8 N/A (queueSize) -1 1 10 100 1000000 -1 1 10 100 1000000 -1 1 10 100 1000000 N/A Score 15,636 15,316 15,425 16,580 15,035 25,945 25,559 25,034 24,971 20,584 24,713 24,265 24,475 24,514 16,595 18,613 Error &pm; 1,905 &pm; 1,237 &pm; 1,673 &pm; 1,133 &pm; 1,148 &pm; 0,939 &pm; 1,250 &pm; 1,414 &pm; 1,010 &pm; 0,655 &pm; 0,687 &pm; 1,022 &pm; 1,137 &pm; 0,836 &pm; 0,739 &pm; 2,040 The actual output of the program is a bit more verbose; it was edited for printing purposes. The Score column shows how many times the benchmark can run in a second. The Error shows that the measurement shows less than 10% scattering. The fastest performance we have is when the algorithm runs on eight threads, which is the number of threads the processor can independently handle on my machine. It is interesting that limiting the size of the queue did not help the performance. I actually expected it to be different. Using a one million length array as a blocking queue has a huge overhead and this is not a surprise that, in this case, the execution is slower than when we have only 100 elements in the queue. The unlimited linked list-based queue handling, on the other hand, fairly fast and clearly shows that the extra speed at the limited queue for 100 elements does not come from the fact that the limit does not allow the IntervalThreads to run too far. When we start one thread, then we expect similar results, as when we run the serial algorithm. The fact that the serial algorithm beats the parallel algorithm running on one thread is not a surprise. The thread creation and the communication between the main thread and the extra one thread have overhead. The overhead is significant, especially when the queue is unnecessarily large. Summary In this chapter, you learned a lot of things. First of all, we refactored the code to be ready for further development that uses parallel guessing. We got acquainted with processes and threads, and we even mentioned fibers. After that, we looked at how Java implements threads and how to create code that runs on multiple threads. Additionally, we saw the different means that Java provides to programmers needing parallel programs, starting threads, or just starting some tasks in already existing threads. Perhaps the most important part of this chapter that you should remember is the metaphor of bureaucrats and the different speeds. This is extremely important when you want to understand the performance of concurrent applications. And I hope that this is a catchy picture, which is easy to remember. There was a huge topic about the different synchronization means that Java provides, and you have also learned about the pitfalls that programmers can fall into when programming concurrent applications. Last but not least, we created the concurrent version of the Mastermind guesser and also measured that it is indeed faster than the version that uses only one processor (at least on my machine). We used the Java Microbenchmark Harness with the Gradle build tool and discussed, a bit, how to perform microbenchmarking. This was a long chapter and not an easy one. I may tend to think that this is the most complex and most theoretical one. If you understood half of it at first read, you can be proud. On the other hand, be aware that this is only a good base to start experimenting with concurrent programming and there is a long way to being senior and professional in this area. And, it is not an easy one. But first of all, be proud of yourself at the end of this chapter. In the following chapters we will learn more about web and web programming. In the very next chapter we will develop our little game so that it can run in a server and the player can play with it using a web browser. This will establish the basic knowledge for web programming. Later we will build on this developing web based service applications, reactive programming and all the tools and areas that will make a professional Java developer. Making Our Game Professional - Do it as a Webapp In this chapter, we will program a web application. We will build on what we have achieved already and create a web version of the Mastermind game. This time, it will not only run alone, guessing and answering the number of positions and matched colors, but also communicate with the user asking for the answers to the guesses. This will be a real game. Web programming is extremely important for Java programmers. Most of the programs are web applications. The universal client available on the Internet is the web browser. The thin-client, web browser-based architecture is widely accepted in enterprises as well. There are only some exceptions when the architecture has something else but the web client. If you want to become a professional Java developer, you must be familiar with web programming. And it is also fun! There are a lot of technical topics that we will visit during the development. First of all, we will discuss networking and web architecture. This is the concrete base of the whole building. It is not too sexy, just like when you construct the building. You spend a lot of money and effort digging trenches, and then you bury the concrete and end up at the end of the phase with what you seemingly had before: flat ground. Except that there is the base. Building without this base, the house would either collapse soon after or during the process of building. Networking is just as important for web programming. There are a lot of topics that seemingly have nothing to do with programming. Still, it is the base of the building and when you program web applications, you will also find the fun part in it. We will also talk a bit about HTML, CSS, and JavaScript, but not too much. We cannot avoid them because they are also important for web programming, but they are topics that you can learn from somewhere else as well. In case you are not an expert in some of these areas, there are usually other experts in enterprise project teams who can extend your knowledge. (In the case of networking, there is no mercy.) In addition to that, JavaScript is a topic so complex and huge that it deserves a whole book to start with it. There are only very few experts who deeply understand both Java and JavaScript. I understand the general structure of the language and the environment it runs in, but I cannot keep up with the new frameworks that are released every week these days, having my focus on other areas. You will learn how to create Java applications that run in an application server, this time in Jetty, and we will see what a servlet is. We will create a web hello world application to start up fast, and then we will create the servlet version of Mastermind. Note that we hardly ever program servlets directly without the aid of some framework that implements the code to handle parameters, authentication, and many other things that are not application-specific. We will still stick to a naked servlet in this chapter because it is not possible to effectively use frameworks, such as Spring, without first understanding what a servlet is. Spring will come in the next chapter. We will mention Java Server Pages (JSP) only because you may meet some legacy application, which was developed using that technology, but modern web applications do not use JSP. Still, JSP is a part of the servlet standard and is available for use. There are other technologies that were developed in the recent past but do not seem to be future-proof these days. They are still usable but appear only in legacy applications, and choosing them for a new project is fairly questionable. We will talk about these technologies shortly in a separate section. By the end of this chapter, you will understand how the basic web technology works and what the major architectural elements are, and you will be able to create simple web applications. This is not enough to be a professional Java web developer but will be a good grounding for the next chapter, where we will have a look at the professional frameworks used in today's enterprises for real application developments. Web and network Programs run on computers, and computers are connected to the Internet. This network was developed in the last 60 years, first to provide military data communication that is resilient to rocket attack, then it was extended to be an academic network, and later it became a commercial network used by anyone and available almost ubiquitously all over the Earth. The design of the network, and the research, started as a response to the flight of Gagarin over the Earth in the fifties. Sending Gagarin to space and travelling over the Earth was a demonstration that Russia could send a rocket anywhere on the globe, possibly with atomic explosives. It meant that any data network that needed some central control was not resilient to such an attack. It was not feasible to have a network with a central location as a single point of failure. Therefore, research was started to create a network that goes on working even if any part of it is brought down. IP The network delivers data packets between any two computers connected to it. The protocol used on the network is IP, which is simply an abbreviation of Internet Protocol. Using IP, a computer can send a data packet to another. The package contains a header and the data content. The header contains the Internet addresses of the sender and the target machine, other flags, and information about the package. Since the machines are not connected to each other directly, routers forward the packets. It is like post offices sending mails to each other till it gets into the hands of the postman you know, who can directly deliver it to your mailbox. To do that, the routers use the information in the header. The algorithm and organization of how the routers interact are complex and something we need not know to be Java professionals. If you ever need to program in order to send IP packets directly, you should look at java.net.DatagramPacket, and the rest is implemented in the JDK, the operating system, and on the firmware of the network card. You can create a data packet; sending it and changing the modulated voltage on the network card or emitting photons to the fiber is not your headache. However, you will all know whether you really need to program datagrams directly. IP has two versions. The old version still in use is IPv4. The new version that coexists with the old one is IPv6 or IPng (ng stands for new generation). The major difference that may concern a Java developer is that version 4 uses 32-bit addresses and version 6 uses 128-bit addresses. When you see a version-4 address, you will see something like 192.168.1.110, which contains the four bytes in a decimal format separated by dots. IPv6 addresses are expressed as 2001:db8:0:0:0:0:2:1, as eight 16-bit numbers expressed in hexadecimal separated by colons. The Web is a bit more complex than sending data packets. If sending a data packet is like sending a onepage letter, then a web page download is like discussing a contract in paper mail. There should be an agreement in the initial paper mail as to what to send, what to answer, and so on, until the contract is signed. On the Internet, that protocol is called Transmission Control Protocol (TCP). While it is highly unlikely (but possible) that you will meet IP routing issues, being a Java developer, you certainly may meet TCP programming. Therefore, we will cover shortly how the TCP works. Be aware that this is very brief. Really. You will not become a TCP expert reading the next section, but you will get a glimpse of the most important issues that affect web programming. TCP/IP The TCP protocol is implemented in the operating system and provides a higher level of interface than IP. When you program TCP, you do not deal with datagrams. Instead, you have a channel of byte streams where you can put bytes to be delivered to the other computer, and you can read bytes from the channel that were sent by the other computer, exactly in the order as they were sent. This is a kind of connection between two computers and, what's more, between two programs. There are other protocols that are implemented over IP and which are not connectionoriented. One of them is User Datagram Protocol (UDP), used for services when there is no need for connections, when the data may be lost and it is more important that the data gets to the destination in a timely manner than losing some of the packets (video streaming, telephony). When the data amount is small and in case it is not delivered, it can be requested again; the cost of losing it is cheap (DNS request, see the next section). When a packet is lost on the network, or when it is sent twice, or when it is delivered sooner than a later package, it is handled by the TCP software layer implemented by the operating system. This layer is also popularly called the TCP stack. Since the TCP is a connected protocol, there is a need for something that tells the TCP stack which stream a datagram belongs to when it arrives. The stream is identified by two ports. A port is a 16-bit integer. One identifies the program that initiates the connection, called the source port. The other one identifies the target program: the destination port. These are contained in each and every TCP packet delivered. When a machine runs a Secure Shell (SSH) server and a web server, they use different ports, usually port 22 and 80. When a package comes that contains the destination port number 22 in the TCP header, the TCP stack knows that the data in the packet belongs to the stream handled by the SSH server. Likewise, if the destination port is 80, then the data goes to the web server. When we program a server, we usually have to define the port number; otherwise, there is no way the clients will find the server program. Web servers are usually listen on port 80, and clients try to connect to that port. The client port is usually not important and not specified; it is allocated by the TCP stack automatically. To connect from a client code to a server is easy: only a few lines of code. Sometimes, it is only one line of code. However, under the hood, there is a lot of work that the TCP stack does that we should care about—it takes time to build up a TCP connection. To have a connection, the TCP stack has to send a datagram to the destination to know that it exists. If there is no server listening on the port, sending the data over the network has no result, except for wasting the network bandwidth. For this reason, the client first sends an empty data packet called SYN. When the other side receives it, it sends back a similar package called SYN-ACK. Finally, the client sends a package called ACK. If the packets go through the Atlantic, this is approximately 45ms for each package, which is equivalent to 45 million seconds in bureaucrat time. This is almost one and a half years. We need three of those to set up the connection, and there is more. When a TCP connection starts, the client does not start to send the data without control. It sends some data packets and then it waits for the server to acknowledge their receipt. It would not only be useless, but also network wasting, to send data that the server is not prepared to accept and has to throw away. The TCP is designed to optimize the network usage. Therefore, the client sends some data, and then it waits for the acknowledgement. The TCP stack automatically manages this. If the acknowledgement arrives, it sends more packets, and if a carefully designed optimization algorithm, implemented in the TCP stack, believes that it is good to send more, it will send a bit more data than in the first step. If there are negative acknowledgements telling the client that the server could not accept some of the data and had to throw it away, then the client will lower the number of packets it sends without acknowledgement. But first it starts slow and cautious. This is called TCP slow start and we have to be aware of it. Although it is a low level networking feature it has consequences that we have to consider in our Java code: we use database connection pools instead of creating a new connection to the database each time there is a need for some data; we try to manage to have as few connections to web servers as possible using techniques such as keep-alive, SPDY protocol, or http/2.0 (also replacing SPDY). For a start, it is enough that TCP is connection-oriented where you build up a connection to a server, send and receive bytes, and finally close the connection. When you have a network performance problem, you have to look at the issues I listed. DNS The TCP protocol creates a channel using the IP addresses of machines. When you type a URL in the browser, it usually does not contain IP numbers. It contains machine names. The name is converted to IP numbers using a distributed database called Domain Name System (DNS). This database is distributed, and when a program needs to convert a name to an address, it sends DNS request to one of the DNS servers it knows. These servers query each other or tell the client whom to ask, until the client knows the IP address assigned to the name. The servers and the client also cache the recently requested names, so answering is fast. On the other hand, when the IP address of a server changes this name, not all clients will immediately see the address assignment over the globe. The DNS lookup can be easily programmed, and there are classes and methods in JDK that support this, but usually we need not care about that; when we program, it is done automatically in web programming. The HTTP protocol The Hypertext Transport Protocol (HTTP) is built on top of the TCP. When you type a URL in a browser, the browser opens a TCP channel to the server (after DNS lookup, of course) and sends a HTTP request to the web server. The server, after receiving the request, produces a response and sends it to the client. After that, the TCP channel may be closed or kept alive for further HTTP request-response pairs. Both the request and the response contain a header and an optional (possibly zero-length) body. The header is in the text format, and it is separated from the body by an empty line. More precisely the header and the body are separated by four bytes: 0x0D, 0x0A, 0x0D, and 0x0A, which are two CR, LF line separators. The HTTP protocol uses carriage return and line feed to terminate lines in the header, and thus, an empty line is two CRLF following each other. The start of the header is a status line plus header fields. The following is a sample HTTP request: GET /html/rfc7230 HTTP/1.1 Host: tools.ietf.org Connection: keep-alive Pragma: no-cache Cache-Control: no-cache Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 DNT: 1 Referer: https://en.wikipedia.org/ Accept-Encoding: gzip, deflate, sdch, br Accept-Language: en,hu;q=0.8,en-US;q=0.6,de;q=0.4,en-GB;q=0.2 The following is the response: HTTP/1.1 200 OK Date: Tue, 04 Oct 2016 13:06:51 GMT Server: Apache/2.2.22 (Debian) Content-Location: rfc7230.html Vary: negotiate,Accept-Encoding TCN: choice Last-Modified: Sun, 02 Oct 2016 07:11:54 GMT ETag: "225d69b-418c0-53ddc8ad0a7b4;53e09bba89b1f" Accept-Ranges: bytes Cache-Control: max-age=604800 Expires: Tue, 11 Oct 2016 13:06:51 GMT Content-Encoding: gzip Strict-Transport-Security: max-age=3600 X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block X-Content-Type-Options: nosniff Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: text/html; charset=UTF-8 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head profile="http://dublincore.org/documents/2008/08/04/dc-html/"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="robots" content="index,follow" /> The request does not contain a body. The status line is as follows: GET /html/rfc7230 HTTP/1.1 It contains the so-called method of the request, the object that is requested, and the protocol version used by the request. The rest of the request of the header contains header fields that have the format, label : value. Some of the lines are wrapped in the printed version, but there is no line break in a header line. The response specifies the protocol it uses (usually the same as the request), the status code, and the message format of the status: HTTP/1.1 200 OK After this, the response fields come with the same syntax as in the request. One important header is the content type: Content-Type: text/html; charset=UTF-8 It specifies that the response body (truncated in the printout) is HTML text. The actual request was sent to the URL, https://tools.ietf.org/html/rfc7230, which is the standard that defines the 1.1 version of HTTP. You can easily look into the communication yourself, starting up the browser and opening the developer tools. Such a tool is built into every browser these days. You can use it to debug the program behavior on the network application level looking at the actual HTTP requests and responses on the byte level. The following screenshot shows how the developer tool shows this communication: HTTP methods The method that is the first word in the status line of the request tells the server what to do with the request. The standard defines different methods, such as GET, HEAD, POST, PUT, DELETE, and some others. The client uses the GET method when it wants to get the content of a resource. In the case of a GET request, the body of the request is empty. This is the method used by the browser when we download a web page. It is also, many times, the method used when some program implemented in JavaScript and running in the browser asks for some information from a web application, but it does not want to send much information to the server. When the client uses POST, the intention is usually to send some data to the server. The server does reply and, many times, there is also a body in the reply, but the main purpose of the request/reply communication is to send some information from the client to the server. This is the opposite of the GET method in some sense. The GET and POST methods are the most frequently used methods. Although there is a general guideline to use GET to retrieve data and POST to send data to the server, it is only a recommendation, and there is no clean separation of the two cases. Many times, GET is used to send some data to the server. After all, it is an HTTP request with a status line and header fields, and although there is no body in the request, the object (part of the URL) that follows the method in the status line is still able to deliver parameters. Many times, it is also easy to test a service that responds to a GET request because you only need a browser and to type in the URL with the parameters, and look at the response in the browser developer tools. You should not be surprised if you see an application that uses GET requests to execute operations that modify the state on a web server. However, not being surprised does not mean approval. You should be aware that in most cases, these are not good practices. When we send sensitive information using the GET request, the parameters in the URL are available to the client in the address line of the browser. When we send using POST, the parameters are still reachable by the client (after all, the information the client sends is generated by the client and, as such, cannot be unavailable), but not that easy for a simple securityunaware user to copy-paste the information and send, perhaps, to a malevolent third party. The decision between using GET and POST should always consider practicalities and security issues. The HEAD method is identical to a GET request, but the response will not contain a body. This is used when the client is not interested in the actual response. It may happen that the client already has the object and wants to see if it was changed. The Last-Modified header will contain the time when the resource was last changed, and the client can decide if it has a newer one or needs to ask for the resource in a new request. The PUT method is used when the client wants to store something on the server and DELETE when the client wants to erase some resource. These methods are used only by applications usually written in JavaScript and not directly by the browser. There are other methods defined in the standard, but these are the most important and frequently used ones. Status codes The response starts with the status code. These codes are also defined and there are a limited number of codes usable in a response. The most important is 200, which says all is OK; the response contains what the request wanted. The codes are always in the range of 100 to 599, contain three digits, and are grouped by the first digit. : These codes are information codes. They are rarely used but can be very important in some cases. For example, 100 means continue. A server can send this code when it gets a POST request and the server wants to signal the client to send the body of the request because it can process it. Using this code, and the client waiting for this code, may save a lot of bandwidth if properly implemented on the server and also on the client. 2xx: These codes mean success. The request is answered properly, or the requested service was done. There are codes, such as 200, 201, 202, and so on, defined in the standard and there is a description about when to use one or the other. 3xx: These codes mean redirection. One of these codes is sent when the server cannot directly service the request but knows the URL that can. The actual codes can distinguish between a permanent redirect (when it is known that all future requests should be sent to the new URL) and temporary redirect (when any later request should be sent here and possibly served or redirected), but the decision is kept on the server side. 4xx: These are error codes. The most famous code is 404, which means Not Found, that is, the server is not able to respond to the request because the resource is not found. 401 means that the resource to serve the request may be available but it requires authentication. 403 is a code that signals that the request was valid but is still refused to be served by the server. 5xx: These codes are server error codes. When a response holds one of these error codes, the meaning is that there is some error on the server. This error can be temporary, for example, when the server is processing too many requests and cannot respond to a new request with a calculationintensive response (this is usually signaled by error code 503) or when the feature is not implemented (code 501). The general error code 500 is interpreted as Internal Error, which means that no information, whatsoever, is available about what was going wrong on the server, but it was not going well and hence, no meaningful response. 1xx HTTP/2.0 After almost 20 years since the last release of HTTP, the new version of HTTP was released in 2015. This new version of the protocol has several enhancements over the previous versions. Some of these enhancements will also affect the way server applications will be developed. The first and most important enhancement is that the new protocol will make it possible to send several resources parallelly in a single TCP connection. The keep-alive flag is available to avoid the recreation of the TCP channel, but it does not help when a response is created slowly. In the new protocol, other resources can also be delivered in the same TCP channel even before one request is fully served. This requires complex package handling in the protocol, but this is hidden from the server application programmer as well as the browser programmer. The application server, servlet container, and browser implement this transparently. HTTP/2.0 will always be encrypted, therefore it will not be possible to use http as a protocol in the browser URL. It will always be https. The feature that will need changes in servlet programming to leverage the advantages of the new version of the protocol is server push. Version 4.0 of the servlet specification includes support for HTTP/2.0, and this version is still in draft. Server push is an HTTP response to a request that will come in the future. How can a server answer a request that is not even issued? Well, the server anticipates. For example, the application sends an HTML page that has references to many small pictures and icons. The client downloads the HTML page, builds the DOM structure, analyzes it, and realizes that the pictures are needed, and sends the request for the pictures. The application programmer knows what pictures are there and may code the server to send the pictures even before the browser requests for it. Every such response includes a URL that this response is for. When the browser wants the resource, it realizes that it is already there and does not issue a new request. In HttpServlet, the program should access PushBuilder via the request's new getPushBuilder method and use that to push down resources to the client. Cookies Cookies are maintained by the browser and are sent in the HTTP request header using the Cookie header field. Each cookie has a name, value, domain, path, expiration time, and some other parameters. When a request is sent to a URL that matches the domain, the path of a non-expired cookie, the client sends the cookie to the server. Cookies are usually stored in small files on the client by the browser or in a local database. The actual implementation is the business of the browser, and we need not worry about it. It is just the text information that is not executed by the client. It is only sent back to the server when some rules (mainly domain and path) match. Cookies are created by servers and are sent to the client in HTTP responses using the Set-Cookie header field. Thus, essentially the server tells the client, Hey, here is this cookie, whenever you come to me next time, show me this piece of information, so I will know it is you. Cookies are usually to remember clients. Advertisers and online shops that need to remember who they are talking to heavily use it. But this is not the only use. These days, any application that maintains user sessions uses cookies to chain up the HTTP requests that come from the same user. When you log in to an application, the username and password you use to identify yourself are sent to the server only once, and in subsequent requests, only a special cookie is sent to the server used to identify the already logged in user. This use of cookies emphasizes why it is important to use cookie values that cannot be easily guessed. If the cookie used to identify a user is easily guessable, then an attacker could just create a cookie and send it to the server mimicking the other user. Cookie values, for the purpose, are usually long random strings. Cookies are not always sent back to the server where they originate. When the cookie is set, the server specifies the domain of the URL where the cookie should be sent back. This is used when a different server from the one providing the services needing authentication does the user authentication. Applications sometimes encode values into cookies. This is not necessarily bad, though in most actual cases, it is. When encoding something into a cookie, we should always consider the fact that the cookie travels through the network and can go huge as more and more data is encoded in it and can create unnecessary burden on the network. Usually, it is better to send only some unique, otherwise meaningless, random key, and store the values in some database, be it on disk or in the memory. Client server and web architecture The applications we developed so far were running on a single JVM. We already have some experience with concurrent programming and this is something that will come handy now. When we program a web application, a part of the code will run on the server and a part of the application logic will execute in the browser. The server part will be written in Java, the browser part will be implemented in HTML, CSS, and JavaScript. Since this is a Java book we will focus mainly on the server part, but we should still be aware of the fact that many of the functionalities can be and should be implemented to run in the browser. The two programs communicate with each other over the IP network, that is, the Internet, or in the case of an enterprise internal application, the network of the company. Today, a browser is capable of running very powerful applications, all implemented in JavaScript. A few years ago, such applications needed client application implemented in Delphi, C++, or Java, using the windowing capabilities of the client operating system. Originally, the client-server architecture meant that the functionality of the application was implemented on the client, and the program was using general services only from the server. The server provided database access and file storage but nothing more. Later, the three-tier architecture put the business functionality on the servers that used other servers for database and other general services, and the client application implemented the user interface and limited business functionality. When the web technology started to penetrate enterprise computing, the web browser started to replace the client applications in many use cases. Previously, the browser could not run complex JavaScript applications. The application was executed on the web server and the client displayed the HTML that the server created as a part of the application logic. Every time something was changed on the user interface, the browser started a communication with the server, and in a HTTP request-response pair, the browser content was replaced. A web application was essentially a series of form filling and form data sending to the server, and the server responded with HTML-formatted pages, presumably containing new forms. JavaScript interpreters were developed and became more and more effective and standardized. Today, modern web applications contain HTML (which is a part of the client code and is not generated by the server on the fly), CSS, and JavaScript. When the code is downloaded from the web server, the JavaScript starts to execute and communicate with the server. It is still HTTP requests and responses, but the responses do not contain HTML code. It contains pure data, usually in the JSON format. This data is used by the JavaScript code and some of the data, if needed, is displayed on the web browser display also controlled by JavaScript. This is functionally equivalent to a three-tier architecture with some slight but very important differences. The first difference is that the code is not installed on the client. The client downloads the application from a web server, and the only thing that is installed is the modern browser. This removes a lot of enterprise maintenance burden and cost. The second difference is that the client is not able, or is limited, to access the resources of the client machine. Thick client applications could save anything in a local file or access a local database. This is very limited, for security reasons, compared to a program running on the browser. At the same time this is a handy limitation because clients aren't and shouldn't be a trusted part of the architecture. The disk in the client computer is hard and expensive to back up. It can be stolen with a notebook, and encrypting it is costly. There are tools to protect client storage, but most of the time, storing the data on the server only is a more viable solution. It is also a common program design error to trust the client application. The client physically controls the client computer and although it can be made technically very difficult, the client can still overcome the security limitations of the client device and client code. If it is only the client application that checks the validity of some functionality or data, then the physical security provided by the physical control of the server is not used. Whenever data is sent from the client to the server, the data has to be checked in regards of validity, no matter what the client application is. Actually, since the client application can be changed, we just don't really know what the client application really is. In this chapter and, as a matter of fact, in the entire book, we focus on Java technologies; therefore the sample application will not contain almost any client technology. I could not help but create some CSS. On the other hand, I definitely avoided JavaScript. Therefore, I have to emphasize again that the example is to demonstrate the programming of the server side and still providing something that really works. A modern application would use REST and JSON communications and would not play around creating HTML on the fly on the server side. Originally, I wanted to create a JavaScript client and REST server application, but the focus was moved so much from server-side Java programming that I dropped this idea. On the other hand, you can extend the application to be one like that. Writing servlets Servlets are Java classes that are executed in a web server that implements the servlet container environment. The first web servers could only deliver static HTML files to the browsers. For each URL, there was an HTML page on the web server and the server delivered the content of this file, in response to a request sent by the browser. Very soon, there was a need to extend the web servers to be able to start some program that calculates the content of the response, on the fly, when the request is processed. The first standard to do that defined CGI. It started a new process to respond to a request. The new process got the request on its standard input, and the standard output was sent back to the client. This approach wastes a lot of resources. Starting a new process, as you learned in the previous chapter, is way too costly just to respond to an HTTP request. Even starting a new thread seems to be unnecessary, but with that, we ran a bit ahead. The next approach was FastCGI, executing the external process continually and reusing it, and then came different other approaches. The approaches after FastCGIall use in-process extensions. In these cases, the code calculating the response runs inside the same process as the web server. Such standards or extension interfaces were ISAPI for the Microsoft IIS server, NSASPI for the Netscape server, and the Apache module interface. Each of these made it possible to create a dynamically loaded library (DLL on Windows or SO files on Unix systems) to be loaded by the web server during startupand to map certain requests to be handled by the code implemented in these libraries. When somebody programs PHP, for example, the Apache module extension is the PHP interpreter that reads the PHP code and acts upon it. When somebody programs ASP pages for the Microsoft IIS, the ISAPI extension implementing the ASP page interpreter is executed (well, this is a bit sloppy and oversimplified to say but works as an example). To Java, the interface definition is a servlet defined in JSR340 as of version 3.1. JSR stands for Java Specification Request. These are requests for modification of the Java language, library interfaces, and other components. The requests go through an evaluation process, and when they are accepted, they become a standard. The process is defined by the Java Community Process (JCP). JCP is also documented and has versions. The current version is 2.10 and can be found at https://jcp.org/en/procedures/overview. The JSR340 standard can be found at https://jcp.org/en/jsr/detail?id=340. A servlet program implements the servlet interface. Usually this is done via extending HttpServlet, the abstract implementation of the Servlet interface. This abstract class implements methods, such as doGet, doPost, doPut, doDelete, doHead, doOption, and doTrace, free to be overridden by the actual class extending it. If a servlet class does not override one of the these methods, sending the corresponding HTTP method, GET, POST, and so on, will return the 405Not Allowed status code. Hello world servlet Before getting into the technical details, let's create an extremely simple hello world servlet. To do it, we setup a Gradle project with the build file, build.gradle, the servlet class in the file, src/main/java/packt/java9/by/example/mastermind/servlet/HelloWorld.java, and last but not least, we have to create the file src/main/webapp/WEB-INF/web.xml. The gradle.build file will look the following: apply plugin: 'java' apply plugin: 'jetty' repositories { jcenter() } dependencies { providedCompile "javax.servlet:javax.servlet-api:3.1.0" } jettyRun { contextPath '/hello' } The Gradle build file uses two plugins, java and jetty. We have already used the java plugin in the previous chapter. The jetty plugin adds tasks such as jettyRun that load the Jetty servlet container and start up the application. The jetty plugin is also an extension of the war plugin that compiles web applications into a Web Archive (WAR) packaging format. The WAR packaging format is practically the same as JAR; it is a zip file and it contains a lib directory that contains all the JAR files that the web application depends on. The classes of the application are in the directory, WEB-INF/classes, and there is a WEB-INF/web.xml file that describes servlet URL mapping, which we will explore in detail soon. Since we want to develop an extremely simple servlet, we add the servlet API as a dependency to the project. This is, however, not a compile dependency. The API is available when the servlet runs in the container. Still, it has to be available when the compiler compiles our code; therefore, a dummy implementation is provided by the artifact specified as providedCompile. Because it is specified that way, the build process will not package the library into the generated WAR file. The generated file will contain nothing that is specific to Jetty or any other servlet container. The servlet container will provide the actual implementation of the servlet library. When the application is deployed and started in a Jetty, the Jetty-specific implementation of the servlet library will be available on the classpath. When the application is deployed to a Tomcat, the Tomcat specific implementation will be available. We create a class in our project, as follows: package packt.java9.by.example.mastermind.servlet; import import import import import import javax.servlet.ServletException; javax.servlet.http.HttpServlet; javax.servlet.http.HttpServletRequest; javax.servlet.http.HttpServletResponse; java.io.IOException; java.io.PrintWriter; public class HelloWorld extends HttpServlet { private String message; @Override public void init() throws ServletException { message = "Hello, World"; } @Override public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html"); PrintWriter out = response.getWriter(); out.println("<h1>" + message + "</h1>"); } @Override public void destroy() { } } When the servlet is started, the init method is invoked. When it is put out of service, the destroy method is called. These methods can be overridden and provide a more fine-grained control than the constructor and other finalization possibilities. A servlet object may be put into service more than once, and after calling destroy, the servlet container may invoke init again; thus, this cycle is not strictly tied to the life cycle of the object. Usually, there is not much that we do in these methods, but sometimes, you may need some code in them. Also, note that a single servlet object may be used to serve many requests, even at the same time; thus, the servlet classes and methods in it should be fairly thread-safe. The specification demands that a servlet container uses only one servlet instance in case the container runs in a non-distributed environment. In case the container runs on the same machine in several processes, each executing a JVM, or even on different machines, there can be many servlet instances that handle the requests. Generally, the servlet classes should be designed such that they do not assume that only one thread is executing them, but at the same time, they should also not assume that the instance is the same for different requests. We just cannot know. What does it mean in practice? You should not use instance fields that are specific to a certain request. In the example, the field initialized to hold the message holds the same value for each and every request; essentially, the variable is almost a final constant. It is used only to demonstrate some functionality for the init method. The doGet method is invoked when the servlet container gets an HTTP request with the GET method. The method has two arguments. The first one represents the request, and the second one represents the response. The request can be used to collect all information that comes in the request. In the preceding example, there is nothing like that. We do not use any of the inputs. If a request comes to our servlet, then we answer the Hello, World string, no matter what. Later, we will see examples when we read the parameters from the request. The response gives methods that can be used to handle the output. In the example, we fetch PrintWriter, which is to be used to send characters to the body of the HTTP response. This is the content that appears in the browser. The mime type we send is text/html, and this is set by calling the setContentType method. This will get into the HTTP header field, Content-Type. The standard and the JavaDoc documentation of the classes define all the methods that can be used, and also how these should be used. Finally, we have a web.xml file that declares the servlets that are implemented in our code. This is, just as the name of the file indicates, an XML file. It declaratively defines all the servlets that are included in the archive and also other parameters. In the example, the parameters are not defined, only the servlet and the mapping to the URL. Since we have only one single servlet in this example, the WAR file, it is mapped to the root context. All and every GET request that arrives to the servlet container and to this archive will be served by this servlet: <?xml version="1.0" encoding="UTF-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <servlet> <display-name>HelloWorldServlet</display-name> <servlet-name>HelloWorldServlet</servlet-name> <servlet-class>packt.java9.by.example.mastermind.servlet.HelloWorld</servlet-class> </servlet> <servlet-mapping> <servlet-name>HelloWorldServlet</servlet-name> <url-pattern>/</url-pattern> </servlet-mapping> </web-app> Java Server Pages I promised you that I would not bore you with Java Server Pages because that is a technology of the past. Even though it is the past, it is still not history as there are many programs running that still use JSP and contain JSP code. JSP pages are web pages that contain HTML and Java code mixed. When an HTTP request is served by a JSP page, the servlet container reads the JSP page, executes the Java parts, takes the HTML parts as they are, and in this way, mixing the two together, creates an HTML page that is sent to the browser. <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%> <html> <body> <% for( int i = 0 ; i < 5 ; i ++ ){ %> hallo<br/> <% } %> </body> </html> The preceding page will create an HTML page that contains the text hallo five times, each in a new line separated by the tag br. Behind the scenes, the servlet container converts the JSP page to a Java servlet, then compiles the servlet using the Java compiler, and then runs the servlet. It does it every time there is some change in the source JSP file; therefore, it is very easy to incrementally craft some simple code using JSP. The code that is generated from the preceding JSP file is 138 lines long (on the Tomcat 8.5.5 version), which is simply long and boring to list here, but the part that may help to understand how the Java file generation works is only a few lines. If you want to see all the lines of the generated servlet class, you can deploy the application into a Tomcat server and look at the directory work/Catalina/localhost/hello/org/apache/jsp/. It is a rarely known fact among developers that this code is actually saved to disk and is available. Sometimes it helps when you need to debug some JSP pages. Here are the few interesting lines generated from the preceding code: out.write("\n"); out.write("<html>\n"); out.write("<body>\n"); for( int i = 0 ; i < 5 ; i ++ ){ out.write("\n"); out.write(" hallo<br/>\n"); } out.write("\n"); out.write("</body>\n"); out.write("</html>\n"); The JSP compiler moves the inside of the JSP code out and the outside in. In the JSP code, Java is surrounded by HTML, and in the generated servlet Java source, the HTML is surrounded by Java. It is like when you want to mend clothes: the first thing is to turn the dress inside out. It is not only the Java code that you can mix into HTML in the JSP pages but also the so-called tags. Tags are collected into tag libraries, implemented in Java, and packaged into JAR files, and they should be available on the classpath to be used. The JSP page using the tags from some library should declare the use: <%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> The tags look like HTML tags, but they are processed by the JSP compiler and executed by the code implemented in the taglib library. JSP may also refer to the value of the Java objects that are available in the scope of the JSP. To do this inside the HTML page, the JSP expression language could be used. JSP was originally created to ease the development of a web application. The main advantage is the fast startup of development. There is no lengthy time for configuration, setup, and so on in the development, and when there is any change in the JSP page, there is no need to compile the whole application again: the servlet container generates the Java code, compiles it to class file, loads the code into memory, and executes. JSP was a competitor of Microsoft ASP pages, which mixed HTML with VisualBasic code. When the application starts to grow big, using the JSP technology causes more problems than are good. The code that mixes the business logic and the view of the application, how it is rendered in the browser, becomes messy. Developing JSP requires frontend technology knowledge. A Java developer is expected to know some frontend technology but is rarely a design expert and CSS guru. Modern code also contains JavaScript, many times embedded in the HTML page. After all, the big advantage of JSP is that it contains code that runs on the server as well as on the client-side code. The developers follow the paradigm many times, so do not be surprised to see some legacy code that contains Java, HTML, CSS, and JavaScript all mixed in a JSP file. Since Java and JavaScript are syntactically similar sometimes, it is not obvious to see what is executed on the server and what is executed on the client. I have even seen code that created JavaScript code from Java code in a JSP file. That is a total mix of different responsibilities and a mess that is nearly impossible to maintain. This led to the total deprecation of JSP as of today. The deprecation of JSP is not official. It is my expert opinion. You may meet some experienced developers who are still in love with JSP, and you may find yourself in projects where you are required to develop programs in JSP. It is not shameful doing that. Some people do worse for money. To mend the messy situation, there were technologies that advocated the separation of the server code and the client functionality more and more. These technologies include Wicket, Vaadin, JSF, and different Java templating engines, such as Freemarker, Apache Velocity, and Thymeleaf. These latter technologies can also be interesting when you generate textual output from Java even when the code is not web-related at all. These technologies, with discipline, helped control the development and maintenance costs of moderate and large web projects, but the basic problem of the architecture was still there: no clear separation of concerns. Today, modern applications implement the code of a web application in separate projects: one for the client, using HTML, CSS and JavaScript, and a separate one to implement server functionality in Java (or in something else, but we focus here on Java). The communication between the two is the REST protocol, which we will cover in the subsequent chapters. HTML, CSS, and JavaScript HTML, CSS, and JavaScript are client-side technologies. These are extremely important for web applications, and a professional Java developer should have some knowledge about them. Nobody expects you to be an expert in Java and in web-client technologies at the same time, though this is not impossible. A certain understanding is desirable. HTML is the textual representation of a structured text. The text is given as characters, as in any text file. Tags represent the structure. A start tag starts with a < character, then the name of the tag, then, optionally, name="value" attributes, and finally a closing > character. An end tag starts with </, then the name of the tag, and then >. Tags are enclosed into hierarchies; thus, you should not close a tag sooner than the one that was opened later. First, the tag that was opened last has to be closed, then the next, and so on. This way, any actual tag in the HTML has a level, and all tags that are between the start and end tags are below this tag. Some tags that cannot enclose other tags or text do not have end tags and stand on their own. Consider the following sample: <html> <head> <title>this is the title</title> </head> </html> The tag head is under html, and title is under head. This can be structured into a tree, as follows: html + head + title + "this is the title" The browser stores the HTML text in a tree structure, and this tree is the object model of the web page document, thus the name, Document Object Model (DOM) tree. The original HTML concept mixed formatting and structure, and even with the current version of HTML5, we still have tags such as b, i, tt that suggest the browser to display the text between the start and end tags in bold, italics, and teletype, respectively. As the name HTML, standing for Hypertext Markup Language, suggests, the text can contain references to other web pages in the form of hyperlinks. These links are assigned to texts using the a tag (standing for anchor) or to some form that may consist of different fields, and when the submit button of the form is pressed, the content of the fields is sent to the server in a POST request. When the form is sent, the content of the fields is encoded in the so-called application/x-www-form-urlencoded form. The HTML structure always tried to promote the separation of structure and formatting. To do so, formatting was moved to styles. Styles defined in Cascading Style Sheets (CSS) provide much more flexibility for formatting than HTML; the format of a CSS is more effective for formatting. The aim to create CSS was that the design can be decoupled from the structure of the text. If I had to choose one of the three, I would opt for CSS as the one that is least important for Java server-side web developers and, at the same time, the most important for the users (things should look nice). JavaScript is the third pillar of client-side technologies. JavaScript is a fully functional, interpreted programming language executed by the browser. It can access the DOM tree, and read and modify it. When the DOM tree is modified, the browser automatically displays the modified page. JavaScript functions can be scheduled and registered to be invoked when some event occurs. For example, you can register a function to be invoked when the document is fully loaded, when the user presses a button, clicks on a link, or just hovers the mouse over some section. Although JavaScript was first only used to create funny animations on the browser, today it is possible, and is a usual practice, to program fully functional clients using the capabilities of the browser. There are really powerful programs written in JavaScript, even such power-hungry applications as PC emulators. In this book, we focus on Java and use the client-side technologies as much as is needed for demonstration technologies. However, being a Java web developer professional, you have to learn these technologies as well, to some extent at least, to understand what a client can do and to be able to cooperate with the professionals responsible for frontend technologies. Mastermind servlet Playing the Mastermind game via the Web is a bit different from what it used to be. Till now, we did not have any user interaction and our classes were designed accordingly. For example, we could add a new guess to the table, along with the partial and full matches calculated by the program. Now we have to separate the creation of a new guess, add it to the game, and set the full and partial matches. This time, we have to display the table first, and the user has to calculate and provide the number of matches. We have to modify some of the classes to be able to do that. We need to add a new method to Game.java: public Row addGuess(Guess guess, int full, int partial) { assertNotFinished(); final Row row = new Row(guess, full, partial); table.addRow(row); if (itWasAWinningGuess(full)) { finished = true; } return row; } Till now, we had only one method that was adding a new guess, and since the program knew the secret, it was immediately calculating the value of full and partial. The name of the method could be addNewGuess, overloading the original method, but this time, the method is used not only to add a new guess but also to add old guesses to rebuild the table. When the program starts, there are no guesses. The program creates one, the first one. Later on, when the user tells the program the full and partial matches, the program needs the Game structure with Table and Row objects containing Guess objects and the full and partial match values. These were already available, but when the new HTTP hit comes in, we have to pull it from somewhere. Programming a servlet, we have to store the state of the game somewhere and restore it when a new HTTP request hits the server. Storing state Storing the state can be done in two places. One place, which we will first do in our code, is the client. When the program creates a new guess, it adds it to the table and sends an HTML page that contains not only the new guess but also all the previous guesses and the full and partial match values that the user gave for each of the rows. To send the data to the server, the values are stored in the fields of a form. When the form is submitted, the browser gathers the information in the fields, creates an encoded string from the content of the fields, and puts the content into the body of a POST request. The other possibility for storing the actual state is in the server. The server can store the state of the game, and it can reconstruct the structure when it creates a new guess. The problem in this case is knowing which game to use. The server can and should store many games, one for each user, and users may use the application concurrently. It does not necessarily mean strong concurrency in the same meaning as we examined in the previous chapter. Even if the users are not served at the same time in multiple threads, there can be games that are active. Imagine cnn.com telling you that you cannot read the news at the moment because somebody else is reading it. There can be multiple users playing multiple games, and while serving an HTTP request, we should know which user we are serving. Servlets maintain sessions that can be used for this purpose as we will see in the next section. HTTP session When a client sends requests from the same browser to the same servlet, the series of requests belong to one session. To know that the requests belong to the same session, the servlet container automatically sends a cookie named JSESSIONID to the client, and this cookie has a long, random, hard-to-guess value (tkojxpz9qk9xo7124pvanc1z as I run the application in Jetty). The servlet maintains a session store that contains the HttpSession instances. The key string that travels in the value of the JSESSIONID cookie identifies the instances. When an HTTP request arrives at the servlet, the container attaches the session to the request object from the store. If there is no session for the key, then one is created, and the code can access the session object by calling the request.getSession() method. A HttpSession object can store attributes. The program can call the setAttribute(String,Object), getAttribute(String), and removeAttribute(String) methods to store, retrieve, or delete an attribute object. Each attribute is assigned to a String and can be any Object. Although the session attribute store essentially looks as simple as a Map<String,?> object, it is not. The values stored in the session can be moved from one node to another when the servlet container runs in a clustered or other distributed environment. To do that, the values are serialized; therefore, the values stored in the session should be Serializable. Failing to do so is a very common novice error. During development, executing the code in a simple development Tomcat or Jetty container practically never serializes the session to disk and never loads it from the serialized form. This means that the values set using setAttribute will be available by calling getAttribute. We run into trouble the first time the application gets installed in a clustered environment. As soon as a HTTP request arrives on different nodes getAttribute may return null. The method setAttribute is called on one node and during the processing of the next request getAttribute on a different node cannot deserialize the attribute value from the disk shared among the nodes. This is usually, and sadly, the production environment. You, as a developer, should also be aware that serializing and de-serializing an object is a heavy operation that costs several CPU cycles. If the structure of the application uses only a part of the client state serving most of the HTTP requests, then this is a waste of CPU to create the whole state in memory from a serialized form and then serializing it again. In such cases, it is more advisable to store only a key in the session and use some database (SQL or NoSQL) or some other service to store the actual data referenced by the key. Enterprise applications almost exclusively use this structure. package packt.java9.by.example.mastermind.servlet; <br/><br/>import packt.java9.by.example.mastermind.Color; <br/>import packt.java9.by.example.mastermind.Table; <br/><br/>import javax.inject.Inject; <br/>import javax.inject.Named; <br/><br/>public class HtmlTools { <br/> @Inject <br/> Table table; <br/><br/> @Inject <br/> @Named("nrColumns") <br/> private int NR_COLUMNS; <br/><br/> public String tag(String tagName, String... attributes) { <br/> StringBuilder sb = new StringBuilder(); <br/> sb.append("<").append((tagName)); <br/> for (int i = 0; i < attributes.length; i += 2) { <br/> sb.append(" "). <br/> append(attributes[i]). <br/> append("=\""). <br/> append(attributes[i + 1]). <br/> append("\""); <br/> } <br/> sb.append(">"); <br/> return sb.toString(); <br/> } <br/> <br/> public String inputBox(String name, String value) { <br/> return tag("input", "type", "text", "name", name, "value", value, "size", "1"); <br/> } <br/><br/> public String colorToHtml(Color color, int row, int column) { <br/> return tag("input", "type", "hidden", "name", paramNameGuess(row, column), <br/> "value", color.toString()) + <br/> tag("div", "class", "color" + color) + <br/> tag("/div") + <br/> tag("div", "class", "spacer") + <br/> tag("/div"); <br/> } <br/><br/><br/> public String paramNameFull(int row) { <br/> return "full" + row; <br/> } <br/><br/> public String paramNamePartial(int row) { <br/> return "partial" + row; <br/> } <br/><br/> public String paramNameGuess(int row, int column) { <br/> return "guess" + row + column; <br/> } <br/><br/> public String tableToHtml() { <br/> StringBuilder sb = new StringBuilder(); <br/> sb.append("<html><head>"); <br/> sb.append("<link rel=\"stylesheet\" type=\"text/css\" href=\"colors.css\">"); <br/> sb.append("<title>Mastermind guessing</title>"); <br/> sb.append("<body>"); <br/> sb.append(tag("form", "method", "POST", "action", "master")); <br/><br/> for (int row = 0; row < table.nrOfRows(); row++) { <br/> for (int column = 0; column < NR_COLUMNS; column++) { <br/> sb.append(colorToHtml(table.getColor(row, column), row, column)); <br/> } <br/><br/> sb.append(inputBox(paramNameFull(row), "" + table.getFull(row))); <br/> sb.append(inputBox(paramNamePartial(row), "" + table.getPartial(row))); <br/> sb.append("<p>"); <br/> } <br/> return sb.toString(); <br/> } <br/>} <html> <br/> <head> <br/> <link rel="stylesheet" type="text/css" href="colors.css"> <br/> <title>Mastermind guessing</title> <br/> <body> <br/> <form method="POST" action="master"> <br/> <input type="hidden" name="guess00" value="3"> <br/> <div class="color3"></div> <br/> <div class="spacer"></div> <br/> <input type="hidden" name="guess01" value="2"> <br/> <div class="color2"></div> <br/> <div class="spacer"></div> <br/> <input type="hidden" name="guess02" value="1"> <br/> <div class="color1"></div> <br/> <div class="spacer"></div> <br/> <input type="hidden" name="guess03" value="0"> <br/> <div class="color0"></div> <br/> <div class="spacer"></div> <br/> <input type="text" <br/> name="full0" value="0" size="1"> <br/> <input type="text" <br/> name="partial0" value="2" size="1"> <br/> <p> <br/> <input type="hidden" name="guess10" value="5"> <br/> <div class="color5"></div> <br/><br/>...deleted content that just looks almost the same... <br/><br/> <p> <br/> <input type="submit" value="submit"> <br/> </form> <br/> </body> <br/> </head> <br/></html> .color0 { <br/> background: red; <br/> width : 20px; <br/> height: 20px; <br/> float:left <br/>} <br/>.color1 { <br/> background-color: green; <br/> width : 20px; <br/> height: 20px; <br/> float:left <br/>} <br/>... .color2 to .color5 is deleted, content is the same except different colors ... <br/><br/>.spacer { <br/> background-color: white; <br/> width : 10px; <br/> height: 20px; <br/> float:left <br/>} Dependency injection with Guice The servlet class is very simple as shown in the following: package packt.java9.by.example.mastermind.servlet; import import import import com.google.inject.Guice; com.google.inject.Injector; org.slf4j.Logger; org.slf4j.LoggerFactory; import import import import import javax.servlet.ServletException; javax.servlet.http.HttpServlet; javax.servlet.http.HttpServletRequest; javax.servlet.http.HttpServletResponse; java.io.IOException; public class Mastermind extends HttpServlet { private static final Logger log = LoggerFactory.getLogger(Mastermind.class); public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { doPost(request, response); } public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { Injector injector = Guice.createInjector(new MastermindModule()); MastermindHandler handler = injector.getInstance(MastermindHandler.class); handler.handle(request, response); } } Because many threads use servlets concurrently, and thus we cannot use instance fields holding data for a single hit, the servlet class does nothing else but create a new instance of a MastermindHandler class and invoke its handle method. Since there is a new instance of MastermindHandler for each request, it can store objects in fields specific to the request. To create a handler, we use the Guice library created by Google. We have already talked about dependency injection. The handler needs a Table object to play, a ColorManager object to manage the colors, and a Guesser object to create a new guess, but creating these or fetching some prefabricated instances from somewhere is not the core functionality of the handler. The handler has to do one thing: handle the request; the instances needed to do this should be injected from outside. This is done by a Guice injector. To use Guice, we have to list the library among the dependencies in build.gradle: apply plugin: 'java' apply plugin: 'jetty' repositories { jcenter() } dependencies { providedCompile "javax.servlet:javax.servlet-api:3.1.0" testCompile 'junit:junit:4.12' compile 'org.slf4j:slf4j-api:1.7.7' compile 'ch.qos.logback:logback-classic:1.0.11' compile 'com.google.inject:guice:4.1.0' } jettyRun { contextPath '/hello' } Then we have to create an injector instance that will do the injection. The injector is created with the following line in the servlet: Injector injector = Guice.createInjector(new MastermindModule()); The instance of MastermindModule specifies what to inject where. This is essentially a configuration file in the Java format. Other dependency injector frameworks used, and use, XML and annotations to describe the injection binding and what to inject where, but Guice solely uses Java code. The following is the DI configuration code: public class MastermindModule extends AbstractModule { @Override protected void configure() { bind(int.class) .annotatedWith(Names.named("nrColors")).toInstance(6); bind(int.class) .annotatedWith(Names.named("nrColumns")).toInstance(4); bind(ColorFactory.class).to(LetteredColorFactory.class); bind(Guesser.class).to(UniqueGuesser.class); } } The methods used in the configure method are created in a fluent API manner so that the methods can be chained one after the other and that the code can be read almost like English sentences. A good introduction to fluent API can be found at https://blog.jooq.org/2012/01/05/the-java-fluent-api-designer-crash-course/. For example, the first configuration line could be read in English as Bind to the class int wherever it is annotated with the @Name annotation having value "nrColor" to the instance 6. (Note that the int value 6 is autoboxed to an Integer instance.) The MastermindHandler class contains fields annotated with @Inject annotation: @Inject @Named("nrColors") private int NR_COLORS; @Inject @Named("nrColumns") private int NR_COLUMNS; @Inject private HtmlTools html; @Inject Table table; @Inject ColorManager manager; @Inject Guesser guesser; This annotation is not Guice-specific. @Inject is a part of the javax.inject package and is a standard part of JDK. JDK does not provide the dependency injector (DI) framework but supports the different frameworks so that they can use standard JDK annotations, and in case the DI framework is replaced, the annotations may remain the same and not framework-specific. When the injector is called to create an instance of MastermindHandler, it looks at the class and sees that it has an int field annotated with @Inject and @Named("nrColors"), and finds in the configuration that such a field should have the value 6. It injects the value to the field before returning the MastermindHandler object. Similarly, it also injects the values into the other fields, and if it should create any of the objects to be injected, it does. If there are fields in these objects, then they are also created by injecting other objects and so on. This way the DI framework removes the burden from the programmers' shoulder to create the instances. This is something fairly boring and is not the core feature of the classes anyway. Instead, it creates all the objects needed to have a functional MastermindHandler and links them together via the Java object references. This way, the dependencies of the different objects (MastermindHandler needs Guesser, ColorManager, and Table; ColorManager needs ColorFactory; and Table also needs ColorManager, and so on) become a declaration, specified using annotations on the fields. These declarations are inside the code of the classes, and it is the right place for them. Where else could we specify what the class needs to properly function than in the class itself? The configuration in our example specifies that wherever there is a need for ColorFactory, we will use LetteredColorFactory, and that wherever we need Guesser, we will use UniqueGuesser. This is separated from the code and it has to be like that. If we want to change the guessing strategy, we replace the configuration and the code should work without modifying the classes that use the guesser. Guice is clever enough and you need not specify that wherever there is a need for Table, we will use Table: there is no bind(Table.class).to(Table.class). First I created a line like that in the configuration, but Guice rewarded me with an error message, and now, writing it again in plain English, I feel really stupid. If I need a table I need a table. Really? The MastermindHandler class We have already started the listing of the MastermindHandler class, and since this class is more than a hundred lines, I will not include it here as a whole. The most important method of this class is handle: public void handle(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { Game game = buildGameFromRequest(request); Guess newGuess = guesser.guess(); response.setContentType("text/html"); PrintWriter out = response.getWriter(); if (game.isFinished() || newGuess == Guess.none) { displayGameOver(out); } else { log.debug("Adding new guess {} to the game", newGuess); game.addGuess(newGuess, 0, 0); displayGame(out); } bodyEnd(out); } We perform three steps. Step 1 is creating the table and we do it from the request. If this is not the start of the game, there is already a table and the HTML form contains all previous guess colors and the answers to those. Then, as the second step, we create a new guess based on that. Step 3 is to send the new HTML page to the client. Again, this is not a modern approach, creating HTML on the servlet code, but demonstrating pure servlet functionality with REST, JSON, and JavaScript with some framework would make this chapter alone a few hundred pages long, and it would definitely distract our focus from Java. Printing HTML text to a PrintWriter is not something that should be new to you at this point in this book; therefore, we will not list that code here. You can download the working example on GitHub. The branch for this version of the code is nosession. Instead of printing, we will focus on the servlet parameter handling. The request parameters are available via the getParameter method, which returns the string value of a parameter. This method assumes that any parameter, be it GET or POST, appears only once in the request. In case there are parameters that appear multiple times, the value should have been a string array. In such a case, we should use getParameterMap, which returns the whole map with the String keys and String[] values. Even though we do not have multiple values for any key this time, and we also know the values of the keys coming as POST parameters, we will still use the latter method. The reason for this is that we will later use the session to store these values, and we want to have a method that is reusable in that case. If you look at the earlier commits in the Git repository, you will see that the first version used getParameter and I refactored it only later when I created the second version of the program, which stores the state in a session. Never believe if anyone tells you that programs are created perfectly upfront without any refactoring during development. Do not feel ashamed to create foolish code and refactor it later. It is only shameful if you do not refactor it. To get to that we convert the request's Map<String,String[]> to Map<String,String>: private Game buildGameFromRequest(HttpServletRequest request) { return buildGameFromMap(toMap(request)); } private Map<String, String> toMap(HttpServletRequest request) { log.debug("converting request to map"); return request.getParameterMap().entrySet(). stream().collect( Collectors.toMap( Map.Entry::getKey, e -> e.getValue()[0])); } Then, we use that map to re-create the game: private Game buildGameFromMap(Map<String, String> params) { final Guess secret = new Guess(new Color[NR_COLUMNS]); final Game game = new Game(table, secret); for (int row = 0; params.containsKey(html.paramNameGuess(row, 0)); row++) { Color[] colors = getRowColors(params, row); Guess guess = new Guess(colors); final int full = Integer.parseInt(params.get(html.paramNameFull(row))); final int partial = Integer.parseInt(params.get(html.paramNamePartial(row))); log.debug("Adding guess to game"); game.addGuess(guess, full, partial); } return game; } The conversion from String to int is done via the method parseInt. This method throws NumberFormatException when the input is not a number. Try to run the game, use the browser, and see how Jetty handles the case when the servlet throws an exception. How much valuable information do you see in the browser that can be used by a potential hacker? Fix the code so that it asks the user again if any of the numbers are not well formatted! Storing state on the server The application state should usually not be saved on the client. There may be some special case in addition to the one where you write educational code and want to demonstrate how to do it. Generally, the state of the application related to the actual use is stored in the session object or on some database. This is especially important when the application requests the user to enter a lot of data and does not want the user to lose the work if there is some hiccup in the client computer. You spend a lot of time selecting the appropriate items in an online shop, choosing the appropriate items that work together, creating a configuration of your new model airplane, and all of a sudden, there is a blackout in your home. If the state were stored on the client you'd have had to start from scratch. If the state is stored on the server, the state is saved to disk; the servers are duplicated, fed by battery-backed power supplies, and when you reboot your client machine when the power comes back in your home, you log in, and miraculously, the items are all there in your shopping basket. Well, it is not a miracle; it is web programming. In our case, the second version will store the state of the game in the session. This will let the user have the game restore so long as the session is there. If the user quits and restarts the browser, the session gets lost and a new game can start. Since there is no need to send the actual colors and matching in hidden fields this time, the HTML generation is modified a bit, and the generated HTML will also be simpler: <html> <head> <link rel="stylesheet" type="text/css" href="colors.css"> <title>Mastermind guessing</title> <body> <form method="POST" action="master"> <div class="color3"></div> <div class="spacer"></div> <div class="color2"></div> <div class="spacer"></div> <div class="color1"></div> <div class="spacer"></div> <div class="color0"></div> <div class="spacer"></div>0 <div class="spacer"></div>2<p> <div class="color5"></div> ... <div class="spacer"></div> <div class="color1"></div> <div class="spacer"></div> <input type="text" name="full2" value="0" size="1"><input type="text" name="partial2" value="0" size="1"> <p> <input type="submit" value="submit"> </form> </body> </head></html> The number of full and partially matching colors is displayed as a simple number, so this version does not allow cheating or changing previous results. (These are the numbers 0 and 2 after the div tags that have the CSS class spacer.) The handle method in MastermindHandler also changes, as shown in the following: public void handle(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { Game game = buildGameFromSessionAndRequest(request); Guess newGuess = guesser.guess(); response.setContentType("text/html"); PrintWriter out = response.getWriter(); if (game.isFinished() || newGuess == Guess.none) { displayGameOver(out); } else { log.debug("Adding new guess {} to the game", newGuess); game.addGuess(newGuess, 0, 0); sessionSaver.save(request.getSession()); displayGame(out); } bodyEnd(out); } This version of the class gets a SessionSaver object injected by the Guice injector. This is a class that we create. This class will convert the current Table into something that is stored in the session, and it also recreates the table from the data stored in the session. The handle method uses the buildGameFromSessionAndRequest method to restore the table and to add the full and partial match answers that the user just gave in the request. When the method creates a new guess and fills it in the table, and also sends it to the client in the response, it saves the state in the session by calling the save method via the sessionSaver object. The buildGameFromSessionAndRequest method replaces the other version, which we named buildGameFromRequest: private Game buildGameFromSessionAndRequest(HttpServletRequest request) { Game game = buildGameFromMap(sessionSaver.restore(request.getSession())); Map<String, String> params = toMap(request); int row = getLastRowIndex(params); log.debug("last row is {}", row); if (row >= 0) { final int full = Integer.parseInt(params.get(html.paramNameFull(row))); final int partial = Integer.parseInt(params.get(html.paramNamePartial(row))); log.debug("setting full {} and partial {} for row {}", full, partial, row); table.setPartial(row, partial); table.setFull(row, full); if (full == table.nrOfColumns()) { game.setFinished(); } } return game; } Note that this version has the same illness of using the parseInt method from the Integer class in JDK, which throws an exception. The GameSessionSaver class This class has three public methods: : This saves a table to the user session restore: This gets a table from the user session reset: This deletes any table that may be in the session save The code of the class is the following: public class GameSessionSaver { private static final String STATE_NAME = "GAME_STATE"; @Inject private HtmlTools html; @Inject Table table; @Inject ColorManager manager; public void save(HttpSession session) { Map<String,String> params = convertTableToMap(); session.setAttribute(STATE_NAME,params); } public void reset(HttpSession session) { session.removeAttribute(STATE_NAME); } public Map<String,String> restore(HttpSession session){ Map<String,String> map= (Map<String,String>) session.getAttribute(STATE_NAME); if( map == null ){ map = new HashMap<>(); } return map; } private Map<String,String> convertTableToMap() { Map<String, String> params = new HashMap<>(); for (int row = 0; row < table.nrOfRows(); row++) { for (int column = 0; column < table.nrOfColumns(); column++) { params.put(html.paramNameGuess(row,column), table.getColor(row,column).toString()); } params.put(html.paramNameFull(row), ""+table.getFull(row)); params.put(html.paramNamePartial(row), ""+table.getPartial(row)); } return params; } } When we save the session and convert the table to a map, we use a HashMap. The implementation in this case is important. The HashMap class implements the Serializable interface; therefore, we can be safe putting it to the session. This alone does not guarantee that everything in HashMap is Serializable. The keys and the values in our case are Strings, and fortunately, the String class also implements the Serializable interface. This way, the converted HashMap object can be safely stored in the session. Also note that, although serialization can be slow, storing HashMap in a session is so frequent that it implements its own serialization mechanism. This implementation is optimized and avoids serialization being dependent on the internal structure of the map. It is time to think about why we have the convertTableToMap method in this class and buildGameFromMap in MastermindHandler. Converting the game and the table in it to a Map and the other way round should be implemented together. They are just two directions of the same conversion. On the other hand, the implementation of the Table to Map direction should use a Map version that is Serializable. This is very much related to session handling. Converting a Map object, in general, to a Table object is one level higher, restoring the table from wherever it was stored: client, session, database, or in the moisture of the cloud. Session storage is only one possible implementation, and methods should be implemented in the class that meets the abstraction level. The best solution could be to implement these in a separate class. You have homework! The reset method is not used from the handler. This is invoked from the Mastermind class, that is, the servlet class to reset the game when we start it: public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { GameSessionSaver sessionSaver = new GameSessionSaver(); sessionSaver.reset(request.getSession()); doPost(request, response); } Without this, playing the game against the machine once would just display the finished game every time we want to start it again, until we exit the browser and restart it or explicitly delete the JSESSIONID cookie somewhere in the advanced menu of the browser. Calling reset does not delete the session. The session remains the same, and thus the value of JSESSIONID too, but the game is deleted from the session object that the servlet container maintains. Running the Jetty web servlet Since we have included the Jetty plugin into our Gradle build, the targets of the plugin are available. To start Jetty is as easy as typing the following: gradle jettyRun This will compile the code, build the WAR file, and start the Jetty servlet container. To help us remember, it also prints the following on the command line: Running at http://localhost:8080//hello We can open this URL and see the opening screen of the game with the colors that the program created as a first guess: Now it is time to have some fun and play with our game, giving answers to the program. Do not make it easy for the code! Refer to the following screenshot: At the same time, if you look at the console where you have typed gradle jettyRun, you will see that the code is printing out log messages, as shown in the following screenshot: These printouts come through the logger that we have in our code. In the previous chapters, we used the System.out.println method calls to send informational messages to the console. This is a practice that should not be followed in any program that is more complex than a hello world Logging There are several logging frameworks available for Java and each has advantages and disadvantages. There is one built into JDK in the java.util.logging package and accessing the logger is supported by the System.getLogger method: the System.Logger and System.LoggerFinder classes. Even though java.util.logging has been available in Java since JDK 1.4, a lot of programs use other logging solutions. In addition to the built-in logging, we have to mention log4j, slf4j and Apache Commons Logging. Before getting into the details of the different frameworks, let's discuss why it is important to use logging instead of just printing to the standard output. Configurability The most important reason is configurability and ease of use. We use logging to record information about the operation of code. This is not the core functionality of the application but is inevitable to have a program that can be operated. There are messages we print out to the log, which can be used by the operating personnel to identify environmental issues. For example, when an IOException is thrown and it gets logged, the operation may look at the logs and identify that the disk got full. They may delete files, or add a new disk and extend the partition. Without the logs, the only information would be that the program does not work. The logs are also used many times to hunt down bugs. Some of the bugs do not manifest in the test environment and are very difficult to reproduce. In such a case, the logs that print out detailed information about the execution of the code are the only source of finding the root cause of some error. Since logging needs CPU, IO bandwidth, and other resources, it should be carefully examined what and when to log. This examination and the decisions could be done during programming, and as a matter of fact, that is the only possibility if we used System.out.println for logging. If we need to find a bug, we should log a lot. If we log a lot, the performance of the system will go down. The conclusion is that we have to log only if it is needed. If there is a bug in the system that cannot be reproduced, the developers ask the operation to switch on debug logging for a short period. Switching on and off different parts of logging is not possible when System.out.println is used. When the debug level log is switched on, the performance may go down for a while, but at the same time, the logs become available for analysis. At the same time, the analysis is simpler when we have to find the log lines that are relevant (and you do not know beforehand which are relevant) if there is a small (a few hundred megabytes log file) rather than a lot of 2-GB compressed log files to find the lines in. Using a log framework, you can define loggers that identify the source of the log messages and log levels. A string usually identifies the logger, and it is a common practice to use the name of the class from which the log message is created. This is such a common practice that the different log frameworks provide factory classes that get the class itself, instead of its name, to get a logger. The possible logging levels may be slightly different in different logging frameworks, but the most important levels are as follows: : This is used when the log message is about some error that prevents the program from continuing its execution. ERROR: This is used when there is some severe error, but the program can still go on functioning although, probably, in some limited manner. WARNING: This is used when there is some condition that is not a direct problem but may later lead to an error if not attended. For example, the program recognizes that a disk is near full, some database connections answer within limits but close to the timeout value, and similar situations. INFO: This is used to create messages about normal operations that may be interesting to operate and are not an error or warning. These messages may help the operation to debug the operational environment settings. DEBUG: This is used to log information about the program that is detailed enough (hopefully) to find a FATAL bug in the code. The trick is that when we put the log statement into the code, we do not know what bug it could be. If we knew, we better fixed it. TRACE: This is even more detailed information about the execution of the code. The log frameworks are usually configured using some configuration file. The configuration may limit the logging, switching off certain levels. In a normal operational environment, the first three levels are usually switched on, and INFO, DEBUG, and TRACE are switched on when really needed. It is also possible to switch on and off certain levels only for certain loggers. If we know that the error is certainly in the GameSessionSaver class, then we can switch on the DEBUG level only for that class. Log files may also contain other information that we did not directly code and would be very cumbersome to print to the standard output. Usually, each log message contains the precise time when the message was created, the name of the logger, and, many times, the identifier of the thread. Imagine if you were forced to put all this to each and every println argument; you would probably soon write some extra class to do that. Don't! It has already been done professionally: it is the logger framework. Loggers can also be configured to send the message to different locations. Logging to the console is only one possibility. Logging frameworks are prepared to send messages to files, database, Windows Event Recorder, syslog service, or to any other target. This flexibility, which message to print, what extra information to print, and where to print is reached by separating the different tasks that the logger framework does into several classes following the single responsibility principle. The logger frameworks usually contain loggers that create the logs, formatters that format the message from the original log information, many times, adding information such as thread ID and time stamp, and appenders that append the formatted message to some destination. These classes implement interfaces defined in the logging framework and nothing but the size of the book stops us from creating our own formatters and appenders. When a log is configured, the appenders and formatters are configured, given the class that implements them. Therefore, when you want to send some logs to some special destination, you are not limited to the appenders that are provided by the authors of the framework. There are a lot of independent open-source projects for different logging frameworks providing appenders for different targets. Performance The second reason to use a logging framework is performance. Although it is not good to optimize for performance before we profile the code (premature optimization), using some methodology known to be slow and inserting several lines into our performance-critical code, invoking slow methods is not really professional either. Using a well-established, highly optimized framework in a way that is industry best practice should not be questionable. Using System.out.println sends the message to a stream and returns only when the IO operation is done. Using real logging handles the information to the logger and lets the logger do the logging asynchronously, and it does not wait for completion. It is really a drawback that log information may be lost if there is some system failure, but this is usually not a serious issue considering how rarely that happens and what is on the other side of the wage: performance. What do we lose if there is a missing debug log line when the disk got full, anyway rendering the system unusable? There is one exception to this: audit logging—when some log information about the transactions of the system has to be saved for legal reasons so that the operation and the actual transactions can be audited. In such a case, the log information is saved in a transactional manner, making the log part of the transaction. Because that is a totally different type of requirement, audit logging is not usually done with any of these frameworks. Also, System.out.println is not synchronized and that way different threads may just garble the output. Log frameworks pay attention to this issue. Log frameworks The most widely used logging framework is Apachelog4j. It currently has a second version that is a total rewrite of the first version. It is very versatile and has many appenders and formatters. The configuration of log4j can be in XML or properties file format, and it can also be configured through API. The author of log4j version 1 created a new logging framework: slf4j. This logging library is essentially a façade that can be used together with any other logging framework. Thus, when you use slf4j in a library you develop, and your code is added to a program as a dependency that uses a different logging framework, it is easy to configure slf4j to send the logs to the loggers of the other framework. Thus, the logs will be handled together and not in separate file, which is desirable to decrease the cost of operation. When developing your library code or an application that uses slf4j, there is no need to select another log framework to slf4j. It has its own simple implementation called backlog. Apache Commons Logging is also a façade with its own logging implementation if nothing else fails. The major difference from slf4j is that it is more flexible in configuration and what underlying logging to use, and it implements a run-time algorithm to discover which logging framework is available and is to be used. The industry best practice shows that this flexibility, which also comes with higher complexity and cost, is not needed. Java 9 logging Java 9 includes a facade implementation for logging. The use is very simple and we can expect that logging frameworks will very soon start to support this façade. The fact that this façade is built into the JDK has two major advantage: The libraries that want to log do not need to have any dependency on any logging framework or logging façade any more. The only dependency is the JDK log façade that is there anyway. The JDK libraries that log themselves use this façade and thus they will log into the same log file as the application. If we use the JDK-provided logging façade the start of the ColorManager class will be changed to the following: package packt.java9.by.example.mastermind; import import import import import import javax.inject.Inject; javax.inject.Named; javax.inject.Singleton; java.util.HashMap; java.util.Map; java.lang.System.Logger; import static java.lang.System.Logger.Level.DEBUG; @Singleton public class ColorManager { protected final int nrColors; protected final Map<Color, Color> successor = new HashMap<>(); private Color first; private final ColorFactory factory; private static final Logger log = System.getLogger(ColorManager.class.getName()); @Inject public ColorManager(@Named("nrColors") int nrColors, ColorFactory factory) { log.log(DEBUG,"creating colorManager for {0} colors", nrColors); In this version we do not import the slf4j classes. Instead we import the java.lang.System.Logger class. Note that we do not need to import the System class, because the classes from the java.lang package are automatically imported. This is not true for the classes that are nested classes in the System class. To get access to the logger the System.getLogger static method is called. This method finds the actual logger that is available and returns one for the name that we pass as argument. There is no version of the method getLogger that accepts the class as the argument. If we want to stick to the convention then we have to write ColorManager.class.getName() to get the name of the class or we can write there the name of the class as a string. The second approach has the drawback that it does not follow the change of the name of the class. Intelligent IDEs like IntelliJ, Eclipse, or Netbeans rename the references to classes automatically but they have a hard time when the name of the class is used in a string. The interface System.Logger does not declare the convenience methods error, debug, warning, and so on, that are familiar from other logging frameworks and façade. There is only one method named log and the first argument of this method is the level of the actual log we issue. There are eight levels defined: ALL, TRACE, DEBUG, INFO, WARNING, ERROR, and OFF. When creating a log message we are supposed to use one of the middle six levels. ALL and OFF are meant to be passed to the isLoggable method. This method can be used to check if the actual logging level gets logged or not. For example, if the level is set to INFO then messages sent with DEBUG or TRACE will not be printed. The actual implementation is located by the JDK using the service loader functionality. The log implementation has to be in a module that provides the interface java.lang.System.LoggerFinder via some implementation. In other words the module should have a class that implements the LoggerFinder interface and the module-info.java should declare which class it is using the code: provides java.lang.System.LoggerFinder with packt.java9.by.example.MyLoggerFinder; The MyLoggerFinder class has to extend the LoggerFinder abstract class with the method getLogger. Logging practice The practice of logging is very simple. If you do not want to spend too much time experimenting with different logging solutions and you do not have some special requirement, then simply go with slf4j, add the JAR to the dependency list as a compile dependency, and start using logging in the source code. Since logging is not instance-specific, and loggers implement thread safety, the log objects that we usually use are stored in a static field, and since they are used as long as the class is used, the program running the field is also final. For example using the slf4j façade we can get a logger with the following command: private static final Logger log = LoggerFactory.getLogger(MastermindHandler.class); To get the logger, the logger factory is used, which just creates the logger or returns the one that is already available. The name of the variable is usually log or logger, but do not be surprised if you see LOG or LOGGER. The reason for uppercasing the name of the variable is that some static code analysis checkers treat static final variables as constants, as they really are, and the convention in the Java community is to use uppercase names for such variables. It is a matter of taste; many times log and logger are used in lowercase. To create a log item the methods trace, debug, info, warn, and error create a message with the respective level as the name implies. For example, consider the following line: log.debug("Adding new guess {} to the game", newGuess); It creates a debug message. Slf4j has support for formatting using the {} literal inside strings. This way, there is no need to append the string from small parts, and in case the actual log item is not sent to the log destination, the formatting will not perform. If we use String concatenation in any form to pass a string as an argument, then the formatting will happen even if debug logging is not desired as per the example. The logging methods also have a version that gets only two arguments: a String message and Throwable. In this case, the logging framework will take care of the output of the exception and the stack trace along with it. If you log something in exception handling code, log the exception and let the logger format it. Other technologies We discussed the servlet technology, a bit of JavaScript, HTML, and CSS. When programming in a real professional environment, these technologies are generally used. The creation of the user interface of applications, however, was not always based on these technologies. Older operating system-native GUI applications as well as Swing, AWT, and SWT use a different approach to create UI. They build up the UI facing the user from program code, and the UI is built as a hierarchical structure of components. When web programming started, Java developers had experience with technologies like these and projects created frameworks that tried to hide the web technology layer. One technology worth mentioning is Google Web Toolkit, which implements the server as well as the browser code in Java, but since there is no Java environment implemented in the browsers, it transpiles (converts) the client part of the code from Java to JavaScript. The last release of the toolkit was created two years ago in 2014 and since then Google has released other types of web programming toolkits that support native JavaScript, HTML, and CSS client development. Vaadin is also a toolkit that you may come across. It lets you write GUI code on the server in Java. It is built on top of GWT and is commercially supported. It may be a good choice in case there are developers available who have experience with GUI development in Java but not in web native technologies, and the application does not require special usability tuning on the client side. A typical intranet corporate application can select it as a technology. Java Server Faces (JSF) is a technology that tries to offload the client-side development of the application from the developers providing widgets ready to be used and the server side. It is a collection of several Java Specification Requests (JSR) and there are several implementations. The components and their relations are configured in XML files and the server creates the client native code. In this technology, there is no transpilation from Java to JavaScript. It is more like using a limited but huge set of widgets, limiting the use to those only, and giving up direct programming of the web browser. If one has the experience and knowledge, however, they can create new widgets in HTML, CSS, and JavaScript. There are many other technologies that were developed to support web applications in Java. The modern approach advocated by most of the big players is to develop the server side and the client side using separate toolsets and methodologies, and connect the two using REST communication. Summary In this chapter, you learnt the structure of web programming. This was not possible without understanding the basics of TCP/IP networking, which is the protocol of the Internet. The application level protocol that is used over that is HTTP, currently in a very new version 2.0, which is still not supported by the servlet standard. We created a version of the Mastermind game that, this time, can really be played using the browser and we started it in a development environment using Jetty. We examined how to store the game state and implemented two versions. Finally, we learned the basics of logging and we looked at other technologies. At the same time, we also looked at the dependency injection implementation Guice from Google, and we studied how it works under the hood, and why and how to use it. After this chapter, you will be able to start the development of a web application in Java and will understand the architecture of such a program. You will understand what is under the hood when you start learning how to program web applications using the Spring framework, which hides many of the complexities of web programming. Building a Commercial Web Application Using REST We were playing around till now, but Java is not a toy. We want to use Java for something real and serious, commercial and professional. In this chapter, we will do that. The example is not something that is only interesting to play with, such as Mastermind in the previous three chapters, but rather a real commercial application. Not a real-life application actually. You should not expect anything like that in a book. It would be too long and not educating enough. However, the application that we will develop in this chapter can be extended and can be used as a core for a real-life application in case you decided to do so. In the previous chapter, we created servlets. To do so, we used the servlet specification, and we handimplemented servlets. That is something you will rarely do these days. Instead, we will use a readily available framework, this time, Spring. It is the most widely used framework for Java commercial applications, and I dare say it is the de facto standard. It will do all the tedious work that we had to do (at least to understand and learn how a servlet works) in the previous chapter. We will also use Spring for dependency injection (why use two frameworks when one does it all?), and we will use Tomcat. In the previous chapter, we used Guice as a DI framework and Jetty as a servlet container. They can be a perfectly good choice for some projects. For other projects, other frameworks do better. To have the opportunity to look at different tools in this book, we will use different frameworks even though all the examples could be created by simply using Tomcat and Spring. The commercial application we will develop will be an ordering system targeting resellers. The interface we will provide to the users will not be a web browser; rather, it will be REST. The users will themselves develop applications that communicate with our system and place orders for different products. The structure of the application we will develop will be microservices architecture, and we will use soapUI to test the application, in addition to the standard Chrome developer tool features. The MyBusiness web shop Imagine that we have a huge trading and logistics company. There are tens of thousands of different products on the shelves; hundreds of lorries come to our warehouse bringing new goods, and hundreds of lorries deliver goods to our customers. To manage the information, we have an inventory system that keeps track of the goods every day, hour, and minute to know what we actually have in the warehouse. We serve our customers without humans managing the warehouse information. Formerly, there were phones, fax machines, and even telex, but today, all we use is the Internet and web services. We do not provide a website for our customers. We have never directly served the end users in our imagined business, but these days, we have a subsidiary that we started off as a separate company to do just that. They have a website, and it is totally independent from us. They are just one of our hundreds of registered partners who each use a web service interface to see the products we have, order products, and track the order status. Sample business architecture Our partners are also large companies with automated administration, with several programs running on several machines. We have no interest in their architecture and the technology they use, but we want to integrate their operations. We want to serve them in a way that doesn't require any human interaction for the administration to order goods on either of our sides. To do so, a web service interface is provided that can be utilized no matter what IT infrastructure they use. On our side, as we imagine the example, we recently replaced our monolithic application with microservices architecture, and though there are still some SOAP-based solutions in the system, most of the backend modules communicate using HTTPS and REST protocols. Some of the modules still rely on asynchronous file transfers done on a daily basis using FTP started from a UNIX cron job. The General Ledger system was programmed in COBOL. Fortunately, we do not need to deal with these dinosaurs. All this structure is an imagined setup but a realistic one. I made up and described these parts to give you a picture of how you may see mixed technologies in a large enterprise. What I described here is a very simple setup. There are companies that have more than a thousand software modules in their systems using different technologies and totally different interfaces, all interconnected with each other. This is not because they like the mess, but that is the way it becomes after 30 years of continuous IT development. New technologies come and old technologies fade out. The business changes and you cannot stick to the old technologies if you want to stay competitive. At the same time, you just cannot replace the entire infrastructure instantaneously. The result is that we see in an enterprise fairly old technologies still running and, many times, new technologies. Old technologies get rolled out by time. They do not stay forever, and still, we are surprised sometimes when a dinosaur comes in front of us. What we have to deal with is the two frontend components that we will develop. These are as follows: Product Information Order Placement and Tracking In the following image, you can see the architectural UML diagram of the structure that we look at. The parts we will have interaction with are only the frontend components, but it helps understand the working and their role if we have a bigger picture: Product Information delivers information about a single product, but it can also deliver a list of products based on the query criteria. Order Placement and Tracking provides functions to place an order and also lets the client to query the state of past orders. To provide product information, we need access to the Product Catalog module that holds the actual product details. There can be a lot of other tasks that the Product Catalog does, and that is the reason it is a separate module. It can have, for example, a workflow and approval engine that lets product administrators to enter product data and managers to check and approve the data. Approval is usually a complex process, considering typos and legal questions (we do not want to trade unlicensed drugs, explosives, and so on), and checking the quality and approval state of the source of the goods. Many complex tasks are included that make it a backend module. In large enterprise applications, the frontend systems rarely do anything else other than the very basic functionality of serving the outside parties. But this is good for us; we can focus on the service that we have to deliver. And this is also good for the architecture. It is the same principle as in object-oriented programming: single responsibility. The Product Information module also has to consult with the Access Control module to see if a certain product can be delivered to the actual customer, and with the inventory to see if there is any product left, so we do not offer a product that is out of stock. The Order Placement and Tracking also needs access to Product Inventory and Access Control modules to check whether the order can be fulfilled. At the same time, it also needs services from the Pricing module, which can calculate the price for the order, and from the Logistics module, which triggers the collection of goods from the inventory locations and shipment to the customer. Logistics also has a connection to invoicing, which has a connection to the General Ledger, but these are on the picture only to show that the travel of information does not end there. There are many other modules that run the company, all of which are none of our interest at the moment. Microservices The architecture described in the previous chapter is not a clean microservice architecture. You will never meet one in its pure form in any enterprise. It is more like something that we really meet in a real company moving from monolithic to microservices. We talk about the microservice architecture when the application is developed in the form of many small services that communicate with each other using some simple API, usually over HTTP and REST. The services implement business functionalities and can be deployed independently. Many times, it is desirable that the service deployment is automated. The individual services can be developed using different programming languages, can use different data storage, and can run on different operating systems; thus, they are highly independent of each other. They can be, and usually are, developed by different teams. The important requirement is that they can cooperate; thus, the API one service implements is usable by the other services that build upon it. The microservice architecture is not the Holy Grail of all architectures. It gives different answers to some problems from monolithic architectures, and many times, these answers work better using modern tools. The applications still have to be tested and debugged, performance has to be managed, and bugs and issues have to be addressed. The difference is that testing can be separated along different technologies; debugging may need more network-related work. These may be good, bad, or both at the same time. For the developers, however, the advantage is clear. They can work on a smaller unit independently and can see the result of their work faster. When developing a single module of a monolithic application, the result can be seen when the entire application gets deployed. In the case of a large application, that may be rare. A typical deployment cycle in a large corporate developing monolithic is every few months, say 3, but it is not rare to see the release development twice or even once a year. Developing microservices, the new module can be deployed as soon as it is ready and tested. If you want to read more on microservices, the first and the most authentic source is the article by Martin Fowler (http://www.martinfowler.com/articles/microservices.html). Service interface design We design the two interfaces that we will implement. When we design interfaces, we focus on the functionality first. Formatting and protocol come later. Interfaces generally should be simple and, at the same time, should accommodate the future change. This is a hard problem because we cannot see the future. Business, logistics, and all other experts may see some part of the future: how the world will change and what it will impose on the operation of the company and, especially, on the interface we provide for our partners. The stability of an interface is of utmost importance because the partners are outside entities. We cannot refactor the code they use. When we change a Java interface in our code, the compiler will complain at all the code locations where the change should be followed. In case of an interface that is used outside of our realm, this is not the case. Even if it is only a Java interface that we publish as open source on GitHub, we should be prepared that our users will face issues if we change the library in an incompatible way. In that case, their software will not compile and work with our library. In the case of an ordering system, it means that they will not order from us and we will soon be out of business. This is one of the reasons why interfaces should be simple. Although this is generally true for most of the things in life, it is extremely important for such interfaces. It is tempting to provide convenience features for the partners because they are easy to implement. In the long run, however, these features may become very expensive as they need maintenance, should be kept backward compatible, and, in the long run, may not gain as much as they cost. To access product information, we need two functions. One of them lists certain products and another returns the details of a specific product. If it were a Java API, it would look as follows: List<ProductId> query(String query); ProductInformation byId(ProductId id); Similarly, order placement may look as shown in the following: OrderId placeOrder(Order order); We provide these functionalities in our application via a web service interface and, more specifically, REST using JSON. We will discuss these technologies in a bit more detailed manner along with the Spring framework and Model View Controller design pattern, but first let's look at the product information controller to get some feeling of how our program will look: package packt.java9.by.example.mybusiness.productinformation; import ... @RestController public class ProductInformationController { @Autowired ProductLookup lookup; @RequestMapping("/pi/{productId}") public ProductInformation getProductInformation(@PathVariable String productId) { ProductInformation productInformation = lookup.byId(productId); return productInformation; } @RequestMapping("/query/{query}") public List<String> lookupProductByTitle(@PathVariable String query, HttpServletRequest request) { //to be developed later } } If you compare the code of the servlet with the preceding code, you can see that this is much simpler. We do not need to deal with the HttpServletRequest object, call an API to get a parameter, or create an HTML output and write it to the response. The framework does this. We annotate the @RestController class, telling Spring that this is a controller that utilizes the REST web services; thus, it will by default create a JSON response from the object we return. We do not need to care about the conversion of the object to JSON, although we can if there is really a need. The object will automatically be converted to JSON using the field names used in the class and the field values of the instance we return. If the object contains more complex structures than just plain String, int, and double values, then the converter is prepared for nested structures and the most common data types. To have different code handling and different URLs on the servlet, all we need to do is to annotate the method with @RequestMapping, providing the path part of the URL. The {productId} notation inside the mapping string is readable and easy to maintain. Spring just cuts the value from there and puts it for us in the productId variable, as requested by the @PathVariable annotation. The actual lookup of the product is not implemented in the controller. That is not the function of the controller. The controller only decides what business logic to invoke and what view to use. A part of it is implemented in the framework, and the very small part you can see the preceding code. The business logic is implemented in a service class. An instance of this class is injected to the lookup field. This is also done by Spring. The actual work we have to do is to invoke the business logic, which this time, since we have only one, is fairly easy. Most of these things seem magic without some more details about what the framework does for us. Therefore, before going on, we will have a look at the building blocks: JSON, REST, MVC, and a bit of the Spring framework. JSON JSON stands for JavaScript Object Notation. It is defined on the site, http://www.json.org/. This is a textual notation in the same way as the object literals are defined in JavaScript . An object representation starts with the { character and ends with the } character. The text in between defines the fields of the objects in the form, string : value. The string is the name of the field, and since JSON wants to be language agnostic, it allows any characters to be a part of the name of a field, and thus this string (as well as any string in JSON) should start and end with the " characters. This may seem strange and, many times, when you start working with JSON, it is easy to forget and write { myObject : "has a string" } instead of the correct { "myObject" : "has a string" } notation. Commas separate the fields. You can also have arrays in JSON. They start and end with [ and ] characters, respectively, and they contain comma-separated values. The value in an object field or in an array can be a string, a number, an object, an array, or one of the constants, true, false, and null. Generally speaking, JSON is a very simple notation to describe data that can be stored in an object. It is easy to write using text editors and easy to read, and thus it is easier to debug any communication that uses JSON instead of more complex formats. Ways to convert JSON to a Java object and the other way round, are readily available in libraries that we will use in this chapter. A sample JSON object that describes a product from our sample code is also available in the source code of the program, as follows: {"id":"125","title":"Bar Stool","description":"another furniture","size":[20.0,2.0,18.0],"weight":300.0} Note that the formatting of JSON does not require a new line, but at the same time, this is also possible. Program-generated JSON objects are usually compact and are not formatted. When we edit some object using a text editor, we tend to format the indentation of the fields in the same way as we usually do in Java programming. REST There is no exact definition of the REST protocol. It stands for Representational state transfer, which probably does not mean a thing to someone who has never heard of it. When we program the REST API, we use the HTTP(S) protocol. We send simple requests to the server, and we get simple answers that we program. This way, the client of the web server is also a program (by the way, the browser is also a program) that consumes the response from the server. The format of the response, therefore, is not HTML formatted using CSS and enriched by client-side functionalities by JavaScript, but rather some data descriptive format such as JSON. REST does not set restrictions on the actual format, but these days, JSON is the most widely used. The wiki page that describes REST is available at https://en.wikipedia.org/wiki/Representational_state_transfer. REST interfaces are usually made simple. The HTTP requests almost always use the GET method. It also makes the testing of REST services simple since nothing is easier than issuing a GET request from a browser. POST requests are only used when the service performs some transaction or change on the server, and that way, the request is sending data to the server rather than getting some data. In our application, we will use the GET method to query a list of products and get information about a product, and we will only use POST to order products. The application that serves these requests will run in a servlet container. You have learnt how to create a naked servlet without the use of a framework. In this chapter, we will use the Spring framework, which offloads many of the tasks from the developer. There are many program constructs in servlet programming that are just the same most of the time. They are called boilerplate code. The Spring framework utilizes the Model View Controller design pattern to develop web applications; thus, we will look at it in brief, before discussing Spring in general. Model View Controller Model View Controller (MVC) is a design pattern. Design patterns are programming constructs: simple structures that give some hint on how to solve some specific problems. The term, design pattern was coined and formally described in the book, Design Patterns, Elements of Reusable Object-Oriented Software, written by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. The book defines a design pattern as a structure with a name, a problem, and a solution. The name describes the pattern and gives the vocabulary for the developer community to use when talking about these patterns. It is important that different developers use the same language terminology in order to understand each other. The problem describes the situation, that is, the design problem where the pattern can be applied. The solution describes classes and objects and the relations between them, which contribute to a good design. One of them is MVC, which is suitable for programming web applications but generally for any application that has a user interface. In our case, we do not have a classical user interface because the client is also a program; still, MVC can be and is a good choice to use. The MVC pattern, as the name also indicates, has three parts: a model, a view, and a controller. This separation follows the single responsibility principle, requiring one part for each distinct responsibility. The controller is responsible for handling the inputs of the system, and it decides what model and view to use. It controls the execution but usually does not do any business logic. The model does the business logic and contains the data. View converts the model data to a representation that is consumable by the client. MVC is a well-known and widely used design pattern, and it is directly supported by Spring in such a way that when you create a web application, you program the controller built into the framework, using annotations; thus, you essentially configure it. You can program the view, but it is more likely that you will use one that is built into the framework. You will want to send data to the client in XML, JSON, or HTML. If you are very exotic, you may want to send YAML, but generally, that is it. You do not want to implement a new format that needs to be programmed on the server and, since this is new, also on the client. We create the model, and this time, we also program it. After all, that is the business logic. Frameworks can do many things for us, mainly the things that are the same for most of the applications but for the business logic. Business logic is the code that distinguishes our code from other programs. That is what we have to program. On the other hand, that is what we like to do. Focus on the business code and avoid all boilerplate provided by the framework. Now that we know what JSON, REST, and the general Model View Controller design pattern are, let's look at how these are managed by Spring and how we can put these technologies into action. Spring framework The Spring framework is a huge one with several modules. The first version of the framework was released in 2003, and since then, there have been four major releases delivering new and enhanced features. Currently, Spring is the de facto enterprise framework used, perhaps more widely than the standard EJB 3.0 Spring supports dependency injection, Aspect-Oriented Programming (AOP), persistence for SQL and NoSQL databases in a conventional and Object Relational Mapping way, transactional support, messaging, web programming, and many other features. You can configure it using XML configuration files, annotations, or Java classes. Architecture of Spring Spring is not monolithic. You can use a part of it, or only some of the features. You can include some of the modules of Spring that you need and leave out others. Some modules depend on some others, but Gradle, Maven, or some other build tool handles that. The following image shows you the modules of the Spring framework for version 4: Spring is constantly developing since its first release, and it is still considered as a modern framework. The core of the framework is a dependency injection container similar to the one we saw in the previous chapter. As the framework developed, it also supported AOP and many other enterprise functionalities, such as message oriented patterns and web programming with an implementation of Model View Controller, supporting not only servlets but also portlets and WebSockets. Since Spring targets the enterprise application arena, it also supports database handling in many different ways. It supports JDBC using templates, Object Relational Mapping (ORM), and transaction management. In the sample program, we use a fairly recent module: Spring boot. This module makes it extremely easy to start writing and running applications, assuming a lot of configurations that are usually the same for many programs. It contains an embedded servlet container that it configures for default settings and configures Spring wherever it is possible, so we can focus on the programming aspect rather than on the Spring configuration. Spring core The central element of the core module is the context. When a Spring application starts, the container needs a context in which the container can create different beans. This is very general and true for any dependency injection container. If we programmatically create two different contexts, they may live independent of each other in the same JVM. If there is a bean declared as a singleton so that there should be only one single instance of it, then the container will create a single instance of it for a context when we need one instance. The objects representing the context have a reference to the object that we have already created. If there are multiple contexts, however, they will not know that there is another context in the JVM that already has an instance, and the container will create a new instance of the singleton bean for the other context. Usually, we do not use more than one context in a program, but there are numerous examples of there being many contexts in a single JVM. When different servlets run in the same servlet container, they run in the same JVM separated by the class loader and they may each use Spring. In this case, the context will belong to the servlet and each will have a new context. In the previous chapter, we used Guice. The Spring context is similar to the Guice injector. In the previous chapter, I was cheating a bit because I was programming Guice to create a new injector for each request. This is far from optimal, and Guice provides an injector implementation that can handle servlet environments. The reason for cheating was that I wanted to focus more on the DI architecture essentials, and I did not want to complicate the code by introducing a complex (well, more complex) implementation of the injector. In the Spring context behavior, what it does, is defined by the interface ApplicationContext. There are two extensions of this interface and many implementations. ConfigurableApplicationContext extends ApplicationContext, defining setters, and ConfigurableWebApplicationContext defines methods needed in the web environment. When we program web applications, we usually do not need to interfere directly with the context. The framework configures the servlet container programmatically, and it contains servlets that create the context and invoke our methods. This is all boilerplate code created for us. The context keeps track of the beans created, but it does not create them. To create beans, we need bean factories (at least one). The topmost interface of bean factories in Spring is BeanFactory. The difference between an object and a bean is that a bean factory creates the bean, it is registered in the context, and it has a String name. This way, the program can refer to the bean by the name. Different beans can be configured in several different ways in Spring. The oldest approach is to create an XML file that describes the different beans, specifying the names, the class that has to be instantiated to create the bean, and fields in case the bean needs other beans to be injected for its creation. The motivation behind this approach is that this way, the bean wiring and configuration can be totally independent of the application code. It becomes a configuration file that can be maintained separately. If we have a large application that may work in several different environments, the access to inventory data may be available in multitude ways. In one environment, the inventory is available by calling SOAP services. In another environment, the data is accessible in an SQL database. In the third environment, it can be available in some NoSQL store. Each of these accesses is implemented as a separate class, implementing a common inventory access interface. The application code depends only on the interface, and it is the container that has to provide one or the other implementation. When the configuration of the bean wiring is in XML, then only this XML file is to be edited, and the code can be started with the implementation of the interface that is suitable for that environment. The next possibility is to configure the beans using annotations. Many times, we use beans and Spring not because there are many implementations for a bean functionality, but because we want to separate the creation of the object instance from the functionality. This is a good style: separation of the concerns even if the implementation is single without alternatives. However, in this case, creating the XML configuration is redundant. If there is an interface and a single implementation of it in our code, then why should I specify in an XML that by creating an object with a class that implements that interface, I should use the class that implements that interface? Quite obvious, isn't it? We do not like programming things that can be automated. To signal that a class can be used as a bean, and to possibly provide a name, we can use the @Component annotation. We do not need to provide a name as an argument. In that case, the name will be an empty string, but why have a name if we do not refer to it? Spring scans all the classes that are on the classpath and recognizes the classes annotated, and it knows that they are the candidates to be used for bean creation. When a component needs another bean to be injected, the field can be annotated with @Autowired or @Inject. The @Autowired annotation is a Spring annotation and existed before the @Inject annotation was standardized. If you intend to use your code outside of the Spring container, it is recommended to use standard annotations. Functionally, they are equivalent. In our code, when Spring creates an instance of the ProductInformationController component, it sees that it needs an instance of ProductLookup. This is an interface, and thus, Spring starts to look for some class that implements this interface, creates an instance of it, possibly first creating other beans, and then injects it, setting the field. You can decide to annotate the setter of the field instead of the field itself. In such a case, Spring will invoke the setter even if the setter is private. You can inject dependencies through constructor arguments. The major difference between the setter, field injection, and constructor injection is that you cannot create a bean without dependency in case you use constructor injection. When the bean is instantiated, it should and will have all other beans injected so that it depends on using the constructor injection. At the same time, the dependencies that need to be injected through the setter injection, or directly into a field, could be instantiated later by the container sometime between instantiating the class and readying the bean. This slight difference may not seem interesting or important until your constructor code may become more complex than the simple dependency settings or until the dependencies become complex. In the case of a complex constructor, the code should pay attention to the fact that the object is not fully created. This is generally true for any constructor code, but in the case of beans created by a dependency injection container, it is even more important. Thus, it may be advisable to use constructor injection. In that case, the dependencies are there; if a programmer makes a mistake, forgetting that the object is not fully initialized, and uses it in the constructor or a method that is called from a constructor, the dependency is there. Also, it is clean and well-structured to use the constructor to initialize the dependencies and have those fields declared final. On the other hand, constructor injection has its downsides. If different objects depend on each other and there is a ring in the dependency graph, then Spring will face a hard time if you use constructor dependencies. When class A needs class B and the other way round, as the simplest circle, then neither A nor B can be created without the other if the dependency injection is constructor dependency. In situations like this, the constructor injection cannot be used, and the circle should be broken at least as a single dependency. In situations like this, setter injection is inevitable. Setter injection may also be better when there are optional dependencies. Many times, some class may not need all its dependencies at the same time. Some class may use a database connection or a NoSQL database handle but not both at the same time. Although it may also be a code smell and probably a sign of poor OO design, it may happen. It may be a deliberate decision to do that because the pure OO design would result in too deep object hierarchies and too many classes, beyond the maintainable limit. If such is the situation, the optional dependencies may be better handled using setter injection. Some are configured and set; some are left with default values, usually null. Last but not least, we can configure the container using Java classes in case the annotations are not enough. For example, there are multiple implementations of the ProductLookup interface, as it is, in our code base. (Don't worry if you did not recognize that; I have not told you about that yet.) There is a ResourceBasedProductLookup class that reads properties files from the package and is mainly to test the application, and there is RestClientProductLookup, which is a production-like implementation of the interface. If I have no other configuration than annotating the lookup field with @Autowired, Spring will not know which implementation to use and will reward the use during startup with the following error message: Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enab 2016-11-03 07:25:01.217 ERROR 51907 --- [ restartedMain] o.s.b.d.LoggingFailureAnalysisReporter : *************************** APPLICATION FAILED TO START *************************** Description: Parameter 0 of constructor in packt.java9.by.example.mybusiness.productinformation.ProductInformationController requi - resourceBasedProductLookup: defined in file [/.../sources/ch07/productinformation/build/classes/main/packt/ - restClientProductLookup: defined in file [/.../sources/ch07/productinformation/build/classes/main/packt/jav Action: Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to This is a fairly self-explanatory error message; it tells a lot. Now is the time when we can configure the bean in XML, but at the same time, we can also configure it using Java. Many developers do not get the point the first time. I did not get it either. The whole XML configuration was to separate the configuration from the code. It was to create the possibility that a system administrator changes the configuration and is free to select one or the other implementation of some interface, wiring the application together. Now Spring tells me that it is better to return to the programmatic way? At the same time, I could hear concerns for many years that XML is not really any better than Java code. XML writing is essentially programming, except that the tooling and IDE support is not as good for XML as it is for Java code (the latter developed a lot in recent years, although for Spring XML configuration). To understand the concept of returning to Java code from XML, we have to get back to the pure reason and aim of the XML way of configuration. The main advantage of XML Spring configuration is not that the format is not programmatic but rather that the configuration code is separated from application code. If we write the configuration in Java and keep those configuration classes to the bare minimum, and they stay as they should, then the separation of application versus configuration code still stands. It is only the format of the configuration that we change from XML to Java. The advantages are numerous. One is that the names of the classes are recognized by the IDE as we edit and we can have autocomplete in Java (note that this also works using XML in some of the IDEs utilizing some of the extensions of plugins). In the case of Java, IDE support is ubiquitous. Java is more readable than XML. Well, this is a matter of taste, but most of us like Java more than XML. System administrators can also edit Java code. When they edit the XML configuration, they usually have to extract it from a JAR or WAR file, edit it, and then package the archive again. In the case of Java editing, they also have to issue a gradle war command or something similar. This should not be a showstopper for a system manager who runs Java applications on a server. And again, it is not Java programming. It is only editing some Java code files and replacing some class name literals and string constants. We follow this approach in our sample application code. We have two configuration files in the application: one for local deployment and testing and another for production. The @Profile annotation specifies which profile the configuration should use. The profile, when the code is executed, can be specified on the command line as a system property, as follows: $ gradle -Dspring.profiles.active=local bootRun The configuration class is annotated with @Configuration. The methods that are bean factories are annotated with @Bean: package packt.java9.by.example.mybusiness.productinformation; import ... @Configuration @Profile("local") public class SpringConfigurationLocal { @Bean @Primary public ProductLookup productLookup() { return new ResourceBasedProductLookup(); } @Bean public ProductInformationServiceUrlBuilder urlBuilder(){ return null; } } The bean factory simply returns a new instance of the ResourceBasedProductLookup class that implements the ProductLookup interface. This implementation can be used to run the application for local testing when there are no external services to rely on. This implementation reads the product data from local resource files packaged into the JAR application. The production version of the configuration is not much different, but as it may be expected, there are a few more things to configure: @Configuration @Profile("production") public class SpringConfiguration { @Bean @Primary public ProductLookup productLookup() { return new RestClientProductLookup(urlBuilder()); } @Bean public ProductInformationServiceUrlBuilder urlBuilder(){ return new ProductInformationServiceUrlBuilder( "http://localhost"); } } This version of the ProductLookup service class uses an external REST service to retrieve the data that it will present to the clients. To do so, it needs the URLs of these services. Such URLs should usually be configured. In our example, we implement a solution where these URLs can be computed on the fly. I tried to make up a situation where it may be needed in real life, but all reasoning was contorted and I gave up. The real reason is that, this way, we can see code that contains a bean that needs another bean to be injected. For now, note that the ProductInformationServiceUrlBuilder instance bean is defined in the same way as the ProductLookup bean, and when it has to be injected into the constructor of the ProductLookup bean, its defining bean method is used and not the following expression directly: new ProductInformationServiceUrlBuilder("http://localhost"); The latter may work, but not in all situations and we should not use it. For the reasons, we will return when we discuss AOP with Spring in a subsequent section. Also note that there is no need to define an interface to define a bean. The type that the bean method returns can also be a class. The context will use the method that fits the needed type, and if there are more than one suitable types and the configuration is not precise enough, as we saw, the container will log an error and will not work. In the configuration that serves the local profile, we create a null value for ProductInformationServiceBuilder. This is because we do not need it when we use local testing. Also, if any method from this class is invoked, it will be an error. Errors should be detected as soon as possible; thus, a null value is a good choice. The ProductInformationServiceUrlBuilder class is very simple: package packt.java9.by.example.mybusiness.productinformation; public class ProductInformationServiceUrlBuilder { private final String baseUrl; public ProductInformationServiceUrlBuilder(String baseUrl) { this.baseUrl = baseUrl; } public String url(String service, String parameter) { final String serviceUrl; switch (service) { case "pi": serviceUrl = baseUrl + ":8081/product/{id}"; break; case "query": serviceUrl = baseUrl + ":8081/query/{query}"; break; case "inventory": serviceUrl = baseUrl + ":8083/inventory/{id}"; break; default: serviceUrl = null; break; } return serviceUrl; } } This bean also needs a constructor parameter, and we used a string constant in the configuration. This clearly shows that it is possible to use a simple object to initialize some of the dependencies (what would stop us, it is pure Java after all), but it may hinder the working of some Spring features. Service classes We have two service classes. These classes serve the controllers with data and implement the business logic, no matter how simple they are. One of the service class implementations calls REST-based services, while the other one reads data from properties files. The latter can be used to test the application offline. The one that calls REST services is used in the production environment. Both of them implement the ProductLookup interface: package packt.java9.by.example.mybusiness.productinformation; import java.util.List; public interface ProductLookup { ProductInformation byId(String id); List<String> byQuery(String query); } stores the whole database in a map called products. It is filled from the properties files when one of the service methods is invoked. The private method loadProducts is invoked from each of the service methods when they start, but it loads the data only if it is not loaded yet: ResourceBasedProductLookup package packt.java9.by.example.mybusiness.productinformation.lookup; import... @Service public class ResourceBasedProductLookup implements ProductLookup { private static Logger log = LoggerFactory.getLogger(ResourceBasedProductLookup.class); The class is annotated using @Service. This annotation is practically equivalent to the @Component annotation. This is only an alternative name to the same annotation. Spring also handles the @Component annotation such that if an annotation interface is annotated using the @Component annotation, the annotation can also be used to signal that a class is a Spring component. You can write your own annotation interfaces if you want to signal for better readability that a class is not a simple component but some other special type. For example, start up your IDE and navigate to the source code of the org.springframework.stereotype.Service interface: private ProductInformation fromProperties(Properties properties) { final ProductInformation pi = new ProductInformation(); pi.setTitle(properties.getProperty("title")); pi.setDescription(properties.getProperty("description")); pi.setWeight( Double.parseDouble(properties.getProperty("weight"))); pi.getSize()[0] = Double.parseDouble(properties.getProperty("width")); pi.getSize()[1] = Double.parseDouble(properties.getProperty("height")); pi.getSize()[2] = Double.parseDouble(properties.getProperty("depth")); return pi; } The fromProperties method creates an instance of ProductInformation and fills it from the parameters given in the Properties object. The Properties class is an old and widely used type. Although there are more modern formats and classes, this is still widely used and it is likely that you will meet this class. This is the very reason we use it here. is a simple Data Transfer Object (DTO) that contains no logic, only fields, setters, and getters. It also contains a constant, emptyProductInformation, holding a reference to an instance of the class with empty values. ProductInformation A Properties object is similar to a Map object. It contains String values assigned to String keys. There are methods, as we will see in our examples, that help the programmer to load a Properties object from a socalled properties file. Such a file usually has the .properties extension, and it contains key value pairs in the following format: key=value For example, the 123.properties file contains the following: id=123 title=Book Java 9 by Example description=a new book to learn Java 9 weight=300 width=20 height=2 depth=18 The properties files are used to store simple configuration values and are almost exclusively used to contain language-specific constants. This is a very contorted use because properties files are ISO Latin-1 encoded files, and in case you need to use some special UTF-8 characters, you have to type them using the \uXXXX format or using the native2ascii converter program. You cannot save them simply as UTF-8. Nevertheless, this is the file format used for language-specific strings used for program internationalization (also abbreviated as i18n because there are 18 characters between the starting i and the last n). To get the Properties object, we have to read the files in the project and get them packaged into a JAR file. The Spring class, PathMatchingResourcePatternResolver, helps us in doing so. Gosh, yes, I know! We have to get used to these long names when we use Spring. Anyway, such long and descriptive names are widely used in an enterprise environment and they are needed to explain the functionality of the classes. We declare a map that will contain all the products during the testing: final private Map<String, ProductInformation> products = new HashMap<>(); The key is the product ID, which is a string in our example. The values are the ProductInformation objects that we fill up using the fromProperties method: private boolean productsAreNotLoaded = true; The next field signals that the products are not loaded: Novice programmers usually use the opposite value with the name productsAreLoaded and set to false by default. In that case, the only place where we will read a value will negate the value, or the main branch of the if command becomes the do nothing part. Neither is a best practice. private void loadProducts() { if (productsAreNotLoaded) { try { Resource[] resources = new PathMatchingResourcePatternResolver() .getResources( "classpath:products/*.properties"); for (Resource resource : resources) { loadResource(resource); } } productsAreNotLoaded = false; } catch (IOException ex) { log.error("Test resources can not be read",ex); } } } The getResources method returns all the resources (files) that are on the classpath under the products directory and that have a.properties extension: private void loadResource(Resource resource) throws IOException { final int dotPos = resource.getFilename().lastIndexOf('.'); final String id = resource.getFilename().substring(0, dotPos); Properties properties = new Properties(); properties.load(resource.getInputStream()); final ProductInformation pi = fromProperties(properties); pi.setId(id); products.put(id, pi); } The product ID is given by the name of the file. This is calculated using simple string manipulation, cutting off the extension. The Resource can also provide an input stream that the Properties class's load method can use to load all the properties at once from the file. Finally, we save the new ProductInformation object in the map. We also have a special noProduct list that is empty. This is returned if there is no product for the query when we want to search for products: private static final List<String> noProducts = new LinkedList<>(); The product lookup service just takes a product from the Map and returns it, or if it does not exist, it returns an empty product: @Override public ProductInformation byId(String id) { loadProducts(); if (products.containsKey(id)) { return products.get(id); } else { return ProductInformation.emptyProductInformation; } } The query is a bit more complex. It implements searching for a product by title. Real-life implementations may implement a more complex logic, but this version is for local testing only; thus, the search by title is enough, perhaps even more complex than would be really necessary: @Override public List<String> byQuery(String query) { loadProducts(); List<String> pis = new LinkedList<>(); StringTokenizer st = new StringTokenizer(query, "&="); while (st.hasMoreTokens()) { final String key = st.nextToken(); if (st.hasMoreTokens()) { final String value = st.nextToken(); log.debug("processing {}={} query", key, value); if (!"title".equals(key)) { return noProducts; } for (String id : products.keySet()) { ProductInformation pi = products.get(id); if (pi.getTitle().startsWith(value)) { pis.add(id); } } } } return pis; } The service class that implements the production functionality is much simpler. Strange, but many times the test code is more complex than the production code: package packt.java9.by.example.mybusiness.productinformation.lookup; import ... @Component public class RestClientProductLookup implements ProductLookup { private static Logger log = LoggerFactory.getLogger(RestClientProductLookup.class); final private ProductInformationServiceUrlBuilder piSUBuilder; public RestClientProductLookup( ProductInformationServiceUrlBuilder piSUBuilder) { this.piSUBuilder = piSUBuilder; } The constructor is used to inject the URL builder bean and this is all the auxiliary code the class has. The rest are the two service methods: @Override public ProductInformation byId(String id) { Map<String, String> uriParameters = new HashMap<>(); uriParameters.put("id", id); RestTemplate rest = new RestTemplate(); InventoryItemAmount amount = rest.getForObject( piSUBuilder.url("inventory"), InventoryItemAmount.class, uriParameters); if ( amount.getAmount() > 0) { return rest.getForObject(piSUBuilder.url("pi"), ProductInformation.class, uriParameters); } else { return ProductInformation.emptyProductInformation; } } The byId method first calls the inventory service to see if there are any products on the inventory. This REST service returns a JSON that has the format, { amount : nnn }; thus, we need a class (so simple that we do not list here) that has the int amount field, the setter, and the getter. The Spring RestTemplate provides an easy way to access a REST service. All it needs is the URL template, a type that is used to convert the result, and a Map object with the parameters. The URL template string may contain parameters in the same way as the request mapping in the Spring controllers, the name of the parameter being between the { and } characters. The template class provides simple methods to access REST services. It automatically does marshaling, sending parameters, and un-marshaling, receiving the response. In the case of a GET request, the marshaling is not needed. The data is in the request URL, and the {xxx} placeholders are replaced with the values from the map supplied as a third argument. The unmarshaling is readily available for most of the formats. In our application, the REST service sends JSON data, and it is indicated in the response Content-Type HTTP header. RestTemplate converts the JSON to the type provided as argument. If ever the server decides to send the response in XML, and it will also be indicated in the HTTP header, RestTemplate will handle the situation automatically. As a matter of fact, looking at the code, we cannot tell how the response is encoded. This is also nice because it makes the client flexible and at the same time, we do not need to deal with such technical details. We can focus on the business logic. At the same time, the class also provides configuration parameters in the case of marshaling or some other functionality so that it automatically needs that. You can, for example, provide marshaling methods, though I recommend that you use whatever is available by default. In most cases, when a developer thinks that there is a need for a special version of any of these functions, the original design of their code is flawed. The business logic is very simple. We first ask the inventory if there is any product in stock. If there is (more than zero), then we query the product information service and return the details. If there is none, then we return an empty record. The other service is even simpler. It simply calls the underpinning service and returns the result: @Override public List<String> byQuery(String query) { Map<String, String> uriParameters = new HashMap<>(); uriParameters.put("query", query); RestTemplate rest = new RestTemplate(); return rest.getForObject( piSUBuilder.url("query"), List.class, uriParameters); } } Compiling and running the application We use gradle to compile and run the application. Since the application does not have any specific configuration that would not appear in most similar applications, it is wise to use the Spring boot. The Spring boot makes it extremely simple to create and run a web application. We need a Java standard public static void main method that starts up the application via Spring: package packt.java9.by.example.mybusiness.productinformation; import ... @SpringBootApplication( scanBasePackageClasses = packt.java9.by.example.mybusiness.SpringScanBase.class) public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); } } The method does nothing but start the StringApplication class's run method. It passes the original arguments and also the class that the application is in. Spring uses this class to read the annotation. The @SpringBootApplication annotation signals that this class is a Spring boot application and provides arguments to configure the packages that contain the application. To do so, you can provide the name of the package that contains the classes, but you can also provide a class in the base package that contains all the classes that Spring has to be aware of. You may not be able to use the class version of the annotation parameter because the root package may not contain any class, only sub-packages. At the same time, providing the name of the root package as String, will not reveal any typo or misalignment during compile time. Some IDE may recognize that the parameter is supposed to be a package name, or it may scan the strings of the program for package names when you refactor or rename a package and give you support for that, but this is more heuristics only. It is a common practice to create a placeholder class that does nothing in the root package in case there is no class there. This class can be used to specify scanBasePackageClasses as an annotation parameter instead of scanBasePackages that needs String. In our example, we have an empty interface, SpringScanBase, as a placeholder. Spring scans all the classes that are on the classpath, recognizes the components and field annotations that it can interpret, and uses this knowledge to create beans without configuration when needed. Note that the abstract class, ClassLoader, included in the JDK does not provide any class scanning method. Since Java environments and frameworks can implement their own ClassLoaders, it is possible (but very unlikely) that some implementation does not provide the scanning functionality provided by the URLClassLoader. URLClassLoader is a non-abstract implementation of the class loading functionality and is a part of the JDK just as well as ClassLoader. We will discuss the intricacies of the class loading mechanism in the subsequent chapters. The gradle build file contains the usual things. It specifies the repositories, the plugins for Java, the IDEs, and also for Spring boot. It also specifies the name of the JAR file that it generates during build. The most important part is the dependency list: buildscript { repositories { mavenCentral() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.4.1.RELEASE") } } apply plugin: 'java' apply plugin: 'eclipse' apply plugin: 'idea' apply plugin: 'spring-boot' jar { baseName = 'packt-ch07-microservice' version = '1.0.0' } repositories { mavenCentral() } bootRun { systemProperties System.properties } sourceCompatibility = 1.9 targetCompatibility = 1.9 dependencies { compile("org.springframework.boot:spring-boot-starter-web") compile("org.springframework.boot:spring-boot-devtools") compile("org.springframework:spring-aop") compile("org.springframework:spring-aspects") testCompile("org.springframework.boot:spring-boot-starter-test") } We depend on Spring boot packages, some test packages, AOP support (which we will look at soon), and also on Spring boot devtools. Spring boot devtools make it possible to restart a web application whenever it is recompiled, without restarting the built-in Tomcat server. Suppose, we start the application using the following command line: gradle -Dspring.profiles.active=production bootRun The Gradle starts up the application and whenever it sees that the classes it runs are modified, it reloads them, and we can test the modified application within a few seconds. The -Dspring.profiles.active=production argument specifies that the production profile should be active. To be able to use this command line parameter, we will also need the bootRun{} configuration closure in the build file. Testing the application The application should have unit tests for each and every class it has except, perhaps, for the DTO classes that contain no functionality. The setters and getters are created by the IDE and are not typed in by the programmer, so it is unlikely that there will be any errors in those. If there is some error related to those classes, it is more likely that it is some integration problem that cannot be discovered using unit tests. Since we discussed unit tests in the previous chapters in detail, we will focus more on integration tests and application tests here. Integration test Integration tests are very similar to unit tests, and many times, novice programmers claim they do unit testing when they actually do integration testing. Integration tests drive the code but do not test the individual classes (units) in isolation, mocking everything that the class may use. Rather, they test the functionality of most of the classes that are needed to perform a test. This way, the integration test does test that the classes are able to work together and not only satisfy their own specifications but also ensure that these specifications work together. In integration test, the external world (like external services) and access to database are mocked only. That is because the integration tests are supposed to run on integration servers, in the same environment where the unit tests are executed, and there these external interfaces may not be available. Many times, databases are mocked using in-memory SQL, and external services are mocked using some mock classes. Spring provides a nice environment to execute such integration tests. In our project, we have a sample integration test: package packt.java9.by.example.mybusiness.productinformation; import ... @RunWith(SpringRunner.class) @SpringBootTest(classes = Application.class) @AutoConfigureMockMvc @ActiveProfiles("local") public class ProductInformationControllerTest { @Autowired private MockMvc mockMvc; @Test public void noParamGreetingShouldReturnDefaultMessage() throws Exception { this.mockMvc.perform(get("/pi")).andDo(print()) .andExpect(status().isNotFound()); } @Test public void paramGreetingShouldReturnTailoredMessage() throws Exception { this.mockMvc.perform(get("/pi/123")) .andDo(print()).andExpect(status().isOk()) .andExpect(jsonPath("$.title") .value("Book Java 9 by Example")); } } This is far from being a complete and full-fledged integration test. There are many situations that are not tested, but here it is good as an example. To have all the support for the Spring environment, we have to use the SpringRunner class. The @RunWith annotation is handled by the JUnit framework, all other annotations are for Spring. When the JUnit framework sees that there is a @RunWith annotation and a runner class specified, it starts that class instead of the standard runner. SpringRunner sets up a Spring context for the test and handles the annotations. specifies the applications that we need to test. This helps Spring to read that class and the annotation on that class, identifying the packages to be scanned. @SpringBootTest @AutoConfigureMockMvc tells Spring to configure a mock version of the Model View Controller framework, which can be executed without a servlet container and web protocol. Using that, we can test our REST services without really going to the network. tells Spring that the active profile is local and that Spring has to use the configuration that is denoted by the annotation, @Profile("local"). This is a version that uses the .properties files instead of external HTTP services; thus, this is appropriate for integration testing. @ActiveProfiles The test performs GET requests inside the mocking framework, executes the code in the controller, and tests the returned value using the mocking framework and fluent API in a very readable way. Note that using the properties files and having the service implementation based on properties file is a bit of an overkill. I created this so that it is possible to start up the application interactively without any real backing service. Consider the following command: gradle -Dspring.profiles.active=local bootRun . If we issue the preceding command, then the server starts up using this local implementation. If we only aim for integration testing, then the local implementation of the service classes should be under the test directory and should be much simpler, mainly only returning constant responses for any expected request and throwing errors if any non-expected request comes. Application test Consider the following command: gradle -Dspring.profiles.active=production bootRun If we start up the application issuing the preceding command and fire up the browser to the URL, http://localhost:8080/pi/123, we will get a fat error message on the browser screen. Ouch... It says Internal Server Error, status=500 or something similar. That is because our code wants to connect to the backing services, but we do not have any yet. To have some to test the application on this level, we should create the backing services or at least something that mocks them. The easiest way is to use the soapUI program. The soapUI is a Java program available from https://www.soapui.org/. There is an open source and free version of it, and there is a commercial version. For our purposes, the free version is enough. We can install it in the simplest click-forward way as it has a setup wizard. After that, we can start it up and use the graphical user interface. We create a new test project, Catalog and inventory, and set up two REST mock services in it: Catalog and Inventory, as shown in the following screenshot: We set up, for each of the mock services, requests to be matched and responses. The content of the response is text and can be typed into the text field on the user interface. It is important that we do not forget to set the media type of the response to application/json (the default is XML). Before starting the services, we have to set the port numbers by clicking on the cogwheel to something that is available on the server. Since 8080 is used by the Tomcat server executed by Gradle, and 8082 is used by soapUI to list the mock services that are currently running, I set the catalog to listen on 8081 and inventory on 8083. You can also see these port numbers in the listing of the ProductInformationServiceUrlBuilder class. The soapUI saves the project in an XML file, and it is available for you on GitHub in the project directory. After starting the mock services, the error message disappears from the browser screen when we press refresh: What we see is exactly what we typed into soapUI. If now I change the inventory mock service to return 0 instead of 100, as in the original version, what I get is the following empty record: {"id":"","title":"","description":"","size":[0.0,0.0,0.0],"weight":0.0} The testing even on this level can be automated. Now, we were playing around using the browser and this is something nice. Somehow, I feel I am producing something when there is a program that is really doing something, when I can see that there is some response in the browser window. However, after a while, this becomes boring and testing manually that the application is still working is cumbersome. It is especially boring for those functions that were not changed. The fact is that they do change miraculously many times even when we do not touch the code that influences them. We touch the code that does influence the function except that we are not aware of it. Poor design, poor coding, or maybe we just forgot, but it happens. Regression test is inevitable. Although browser testing user interfaces can also be automated, this time, we are having a REST service that we can test and that is what soapUI is for. We have already installed the tool, we have already started it, and we have some mock services running in it. The next thing is to add a New REST service from URI to the project and specify the URL, http://localhost:8080/pi/{id}, exactly the same way as we did for Spring: When we have a REST service defined in the project, we can create a new Test Suite and a Test Case inside the suite. We can then add a step to the Test Case that will call the REST service using the parameter 123 if we modify the default value, which is the same as the name of the parameter, in this case, id. We can run the test step using the green triangle on the upper-left corner of the window, and since we have the tested application and the soapUI mock services running, we should get an answer in JSON. We have to select JSON on the response side; otherwise, soapUI tries to interpret the response as XML, and since we have a JSON response, it is not too fruitful. What we see is the following window: It is the same response that we saw in the browser. There are no miracles when we program computers. Sometimes, we do not understand what happens, and some things are so complex that they seem to be a miracle, but they are actually not. There is an explanation for everything, it may just not be known to us. In this case, we certainly know what is happening, but why is it any better to see the JSON on the screen of soapUI than it is on the browser? The reason is that soapUI can execute assertions and in some cases, further test steps based on the result of the REST invocation, and the final result is a simple YES or NO. The test is OK, or it FAILS. To add an assertion, click on the Assertions text on the lower-left corner of the window. As you can see in the preceding screenshot, I have already added one that compares the "title" field of the returned JSON with the text "Bar Stool". When we add the assertion, the default value it suggests is the one that was actually returned, which is just a very handy feature. After this, running the whole test suite again will run all the test cases (we have only one), and all the test steps, one after the other (we again have only one), and finally it will display a green FINISHED bar on the UI, as shown in the following screenshot: This is not all that soapUI can do. This is a well-developed test tool that has been in the market for many years. soapUI can test SOAP services and REST services, and it can handle JMS messages. You can create tests of many steps with these calls, loops, and assertions in calls or in separate tests, and in case all else fails, you can do just anything by creating programmed steps in the Groovy language or creating extensions in Java. Servlet filters The services work fine by now and anyone can query the details of our products. That may be a problem. The details of the products are not necessarily public information. We have to ensure that we serve the data only to partners who are eligible to see it. To ensure that, we need something in the request that proves that the request comes from a partner. This information is typically a password or some other secret. It could be placed into the GET request parameters or into the HTTP request header. It is better to put it into the header because the information is secret and not to be seen by anybody. The GET parameters are a part of the URL, and the browser history remembers that. It is also very easy to enter this information into the browser location window, copy paste it, and send it over a chat channel or over e-mail. This way, a user of the application, who is not so educated and concerned about security, may disclose secret information. Although it is not impossible to do the same with information that is sent in an HTTP header, it is not likely to happen. If the information is in the header and somebody sends the information in an e-mail, they probably know what they are doing; they cross a security border willingly and not by simple negligence. To send authentication information along the HTTP request, Spring provides a security module that can easily be configured with annotations and configuration XMLs and/or classes. This time, we will do it a bit differently to introduce servlet filters. We will require that the vendors insert the X-PartnerSecret header into the request. This is a non-standard header, and thus it must have the X- prefix. Following this approach is also some extra security feature. This way, we can prevent the user from reaching the service using a simple browser. There is, at least, a need for some extra plugin that can insert a custom header or some other program such as soapUI. This way, it will ensure that our partners will use the interface programmatically, or if ever they need to test the interface ad hoc, only users with a certain level of technology can do so. This is important to keep the support costs controlled. Since this secret has to be checked in the case of each and every service, we better not insert the checking code into each and every service controller. Even if we create the code properly and factor the check for the secret into a separate class, the invocation of the method asserting that the secret is there and is correct will have to be inserted in each and every controller. The controller does the service; checking the client authenticity is an infrastructure issue. They are different concerns, and thus, they have to be separated. The best way that the servlet standard provides for us is a servlet filter. A servlet filter is a class invoked by the servlet container before the servlet itself if the filter is configured. The filter can be configured in the web.xml configuration file of the servlet container or by using an annotation when we use the Spring boot. The filter does not only get the request and response as parameters but also a third argument of the FilterChain type that it should use to call the servlet or the next filter in the chain. There can be more than one filter defined and they get chained up. The filter may, at its discretion, decide to call or not to call the next in the chain. We put our servlet filter into the auth sub-package of our application: package packt.java9.by.example.mybusiness.productinformation.auth; import ... @Component public class AuthFilter implements Filter { private static Logger log = LoggerFactory.getLogger(AuthFilter.class); public static final int NOT_AUTHORIZED = 401; @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest httpRequest = (HttpServletRequest) request; final String secret = httpRequest.getHeader("X-PartnerSecret"); log.info("Partner secret is {}", secret); if ("packt".equals(secret)) { chain.doFilter(request, response); } else { HttpServletResponse httpResponse = (HttpServletResponse) response; httpResponse.sendError(NOT_AUTHORIZED); } } @Override public void destroy() { } } The filter implements the Filter interface that defines three methods. In our case, we do not have any parameters to consider in the filter, and we do not allocate any resources to release; thus, both init and destroy methods are empty. The main work of the filter is the doFilter method. It has three parameters, two of them are the same as the parameters of a servlet and the third is FilterChain. The request is converted to HttpServletRequest, so we can get access to the X-PartnerSecret header through the getHeader method. If the value sent in this header field is good, we call the next in the chain. In our application, there are no more filters configured; therefore, the next in the chain is the servlet. If the secret is not acceptable, then we do not call the next in the chain. Instead, we return the 401 Not Authorized HTTP error to the client. In this application, the secret is very simple. This is the constant string packt. This is not really a big secret, especially now that it is published in this book. A real-life application would require something more cryptic and less known. It is very probable that each partner would use different secrets and that the secret has to change from time to time. When there is an error condition in a servlet that our program handles, it is a good practice to use the HTTP error handling mechanism. Instead of sending back a message with the status code 200 OK and explaining, for example, in a JSON format that the authentication was not successful, we have to send back the 401 code. This is defined by the standard and does not need any further explanation or documentation. There is one thing left in our program, and that is audit logging. Audit logging and AOP We have logging in our sample code and for that we use slf4j, which we covered in the previous chapter. Logging is more or less the decision of the developer and supports technical levels of operation. There, we also touched on a few sentence audit loggings. This type of logging is usually explicitly required in a functional requirement. Generally, AOP is separating the different aspects of code functionality into separate code fragments, and implementing them independent of each other. This is very much the single responsibility principle. This time, it is implemented in a way that not only the different functionalities are implemented separately but also how we connect them together is defined separately. What is executed before and after what other parts are encoded separately gets to the Spring configuration. We have seen something similar already. The dependencies that a class needs to properly operate are defined in a separate segment (XML or Java code). It is not a surprise that in the case of AOP, the same is done using Spring. Aspects are configured in the configuration file or class. A typical aspect is audit logging, and we will use this as an example. There are many topics that can be implemented using aspects, and some of them are even worth implementing that way. We do not want to implement the audit logging code in each business method or class that needs it. Instead, we implement a general aspect and configure the wiring such that whenever a bean method that needs audit logging is invoked, Spring invokes the audit logging. There are other important terminologies that we should understand for AOP and especially how AOP can be configured in Spring. The first and most important is the aspect. This is the functionality that we want to implement, in our example, the audit logging. Join point is the point in execution when an aspect is invoked. When using a full-scale aspect solution in Java that modifies the byte code of the generated class, a join point can be almost anything. It can be access to a field, read or write; it can be the invocation of a method or exception throwing. In the case of Spring, the class byte code is not modified; thus, Spring is not able to identify the access of a field or an exception throwing. Using Spring, a join point is always used when a method is invoked. An advice is how the aspect is invoked at the join point. It can be before advice, after advice, or around advice. When the advice is before, the aspect is invoked before the method is called. When the advice is after, the aspect is invoked after the method is invoked. Around means that the aspect is invoked before the method call, and the aspect also has an argument to call the method and still perform some actions after the method is called. This way, the around advice is very similar to servlet filters. The before advice is called before the method call, and after it returns, the framework will invoke the method. There is no way for the aspect to prevent the invocation of the original method. The only exception is when the aspect, well, throws an exception. The after advice is also affected by exceptions. There can be an after returning advice that is invoked when the method is returning. The after throwing is invoked only if the method were throwing an exception. After finally is invoked in the case of an exception or return. Pointcut is a special string expression that identifies join points. A pointcut expression may match zero, one, or more join points. When the aspect is associated with a pointcut expression, the framework will know the join points and when and where to invoke the aspect. In other words, pointcut is the string that tells when and for which method to invoke the aspect. Even though Spring implementation of AOP does not use AspectJ and does not modify the byte code that was created for the classes, it supports the pointcut expression language. Although this expression language provides more features than what Spring implements, it is a well-established and widely used and accepted expression language to describe pointcuts, and it just would not make sense to invent something new. Introduction is adding methods or fields to a type that already exists and doing it during runtime. Spring allows this AOP functionality to add an interface to an existing type and add an implementation of the interface in the form of an advice class. In our example, we do not use this functionality. Target object is the object that is being advised by the aspect. This is the bean that contains the method around the aspect, that is, before or after the aspect is invoked. That was just a condensed set of definitions, almost like in a math book. If you did not get the point just reading it, don't worry. I did not understand it either. That is why we have the following example, after which all we just covered will make more sense: package packt.java9.by.example.mybusiness.productinformation; import ... @Configuration @Aspect public class SpringConfigurationAspect { private static Logger log = LoggerFactory.getLogger("AUDIT_LOG"); @Around("execution(* byId(..))") public ProductInformation byIdQueryLogging( ProceedingJoinPoint jp) throws Throwable { log.info("byId query is about to run"); ProductInformation pi = (ProductInformation) jp.proceed(jp.getArgs()); log.info("byId query was executed"); return pi; } @Around("execution(* url(..))") public String urlCreationLogging(ProceedingJoinPoint jp) throws Throwable { log.info("url is to be created"); String url = (String) jp.proceed(jp.getArgs()); log.info("url created was "+url); return url; } } The class is annotated with the @Configuration annotation so that Spring knows that this class contains the configuration. The @Aspect annotation denotes that this configuration may also contain aspect definitions. The @Around annotation on the methods gives the type of advice, and the argument string for the annotation is the pointcut expression. If the type of advice is different, one of the annotations, @Before, @After, @AfterReturning, or @AfterThrowing, should be used. In our example, we use the @Around aspect to demonstrate the most complex scenario. We log the execution of the target method before and after the execution of the method, and we also call the original method through the ProceedingJoinPoint object. Because the two objects return different types and we want to log differently, we define two aspect methods. The argument of the advice annotation is the pointcut string. In this case, it is a simple one. The first one, execution(* byId(..)), says that the aspect should be invoked for any execution of any method that has the name byId and has any arguments. The second is very similar, except the name of the method is different. These are simple pointcut expressions, but in a large application that heavily uses AOP, they can be very complex. The pointcut expression syntax in Spring mainly follows the syntax used by AspectJ. The expression uses the notion of point cut designator (PCD) that is usually execution. It is followed by the pattern that defines which method to intercept. The general format is as follows: execution(modifiers-pattern? ret-type-pattern declaring-type-pattern?name-pattern(param-pattern) throws-pattern?) Except for the return type part, all other parts are optional. For example, we can write the following: execution(public * *(..)) This will intercept all public methods. The following expression intercepts all methods that have a name starting with set: execution(* set*(..)) We can use the * character as a joker in the same way as we can use it on the command line in Windows or Unix shell. The argument matching definition is a bit more complex. (..) means any arguments, () means no arguments, and (*) means exactly one argument of any type. The last one can also be used when there are more arguments; for example, (*,Integer) means that there are two arguments, the second being an Integer, and we just do not care what the type of the first one is. Pointcut expressions can be more complex, joining together match expressions with the && (and) and || (or) logical operators, or using the ! (negation) unary operator. Using the @Pointcut() annotation, the configuration can define pointcuts putting the annotations on methods. For example, consider the following: @Pointcut("execution(* packt.java.9.by.example.service.*.*(..))") public void businessService() {} It will define a join point for any method that is implemented in any class in the packt.java.9.by.example.service package. This merely defines the pointcut expression and assigns it to the name businessService, which is given by the name of the method. Later, we can refer to this expression in aspect annotations, for example: @After("businessService()") Note that the use of the method is purely for its name. This method is not invoked by Spring. It is only used to borrow its name to the expression that is defined on it using the @Pointcut annotation. There is a need for something, such as a method, to put this annotation on, and since methods have names, why not use it: Spring does it. When it scans the configuration classes and sees the annotation, it assigns it in its internal structures to the name of the method, and when that name (along with the parenthesis, to confuse the novice programmer mimicking a method call) is used, it looks up the expression for that name. AspectJ defines other designators. Spring AOP recognizes some of them, but it throws IllegalArgumentException because Spring implements only method execution pointcuts. AspectJ, on the other hand, can also intercept object creation for which the PCD is initialization, as an example. Some other PCDs, in addition to execution, can limit an execution PCD. For example, the PCD, within, can be used to limit the aspect to join points belonging to classes within certain packages, or the @target PCD can be used to limit the matching to methods in objects that have the annotation given between ( and ) after the keyword @target in the pointcut expression. There is a PCD that Spring uses that does not exist in AspectJ. This is a bean. You can define a pointcut expression that contains bean(name pattern) to limit the join point to method executions that are in the named bean. The pattern can be the entire name or it can be, as almost any PCD expression matching, * as a joker character. Dynamic proxy-based AOP Spring AOP, when first presented to Java programmers, seems like magic. How does it happen that we have a variable of classX and we call some method on that object, but instead, it executes some aspect before or after the method execution, or even around it, intercepting the call The technique that Spring does is called dynamic proxy. When we have an object, which implements an interface, we can create another object—the proxy object—that also implements that interface, but each and every method implementation invokes a different object called handler, implementing the JDK interface, InvocationHandler. When a method of the interface is invoked on the proxy object, it will call the following method on the handler object: public Object invoke(Object target, Method m, Object[] args) This method is free to do anything, even calling the original method on the target object with the original or modified argument. When we do not have an interface at hand that the class to be proxied implements, we cannot use JDK methods. Luckily, there are widely used libraries, such as cglib, which are also used by Spring and that can do something similar. Cglib can create a proxy object that extends the original class and implements its methods, invoking the handler object's invoke method in a way similar to how the JDK version does for the interface methods. These technologies create and load classes into the Java memory during runtime, and they are very deep technical tools. They are advanced topics. I do not say not to play with them while being a novice Java programmer. After all, what can happen? Java is not a loaded gun. It is, however, important that you do not lose your interest when you do not understand some of the details or something does not work first. Or second. Or third... Keep swimming. AOP implementation in Spring works by generating proxy objects for the target objects, and the handlers invoke the aspects that we define in the Spring configuration. This is the reason you cannot put aspects on final classes or on final methods. Also, you cannot configure aspects on private or protected methods. The protected methods could be proxied in principle, but this is not a good practice, and thus Spring AOP does not support it. Similarly, you cannot put aspects on classes that are not Spring beans. They are created by the code directly and not through Spring and have no chance to return a proxy instead of the original object when the object is created. Simply put, if Spring is not asked to create the object, it cannot create a custom one. The last thing we want to do is to execute the program and see how the aspects perform. The implementation of our audit logging is extremely simple. We use the standard logging, which is not really sufficient for a real-life application of audit logging. The only special thing we do is that we use a logger identified by the name, AUDIT_LOG and not by the name of a class. This is a legitimate use of the loggers in most of the logging frameworks. In spite of the fact that we usually use the class to identify the logger, it is absolutely possible to use a string to identify a logger. In the case of our logging, this string will also be printed on the console in the log lines, and it will visually stand out. Consider the following command: gradle -Dspring.profiles.active=production bootRun If we again start the application with the preceding command, start soapUI for the project, start the mock services, and execute the test, we will see on the console the following log lines that the aspects print: 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 2016-11-10 19:14:09.559 19:14:09.567 19:14:09.626 19:14:09.629 19:14:09.655 19:14:09.666 19:14:09.691 19:14:09.715 19:14:09.716 19:14:09.716 19:14:09.716 19:14:09.725 INFO INFO INFO INFO INFO INFO INFO INFO INFO INFO INFO INFO 74643 74643 74643 74643 74643 74643 74643 74643 74643 74643 74643 74643 ------------------------- [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] o.s.web.servlet.DispatcherServlet o.s.web.servlet.DispatcherServlet p.j.b.e.m.p.auth.AuthFilter AUDIT_LOG AUDIT_LOG AUDIT_LOG p.j.b.e.m.p.l.RestClientProductLookup p.j.b.e.m.p.l.RestClientProductLookup AUDIT_LOG AUDIT_LOG AUDIT_LOG : : : : : : : : : : : : Initializing Spr FrameworkServlet FrameworkServlet Partner secret i byId query is ab url is to be cre url created was amount {id: 123, There items from url is to be cre url created was byId query was e Summary In this chapter, we built a simple business application that supports business-to-business transactions. We implemented a REST service in a microservices (almost) architecture using the features that are provided by the de facto standard enterprise framework: Spring. Looking back at the chapter, it is amazing how few lines of code we wrote to achieve all the functionality, and that is good. The less code we need to develop what we want, the better. This proves the power of the framework. We discussed microservices, HTTP, REST, JSON, and how to use them using the MVC design pattern. We learned how Spring is built up, what modules are there, how dependency injection works in Spring, and we even touched a bit of AOP. This was very important because along with AOP, we discovered how Spring works using dynamic proxy objects, and this is something that is very valuable when you need to debug Spring or some other framework that uses a similar solution (and there are a few frequently used). We started to test our code using a simple browser, but after that we realized that REST services are better tested using some professional testing tool, and for that we used soapUI and built up a simple REST test suite with REST test steps and mock services. Having learnt all that, nothing will stop us from extending this application using very modern and advanced Java technologies, such as reflection (which we have already touched on a bit when we discussed the JDK dynamic proxy), Java streams, lambda expressions, and scripting on the server side. Extending Our E-Commerce Application In the last chapter, we started developing an e-commerce application and we created the functionality to look up products based on their ID and, also, by some parameters. In this chapter, we will extend the functionality so that we can also order the products we selected. While doing so, we will learn new technologies, focusing on functional programming in Java and on some other language features, such as reflection and annotation handling during runtime, and scripting interface. As we did in the previous chapters, we will develop the application step by step. As we discover the newly learnt technologies, we will refactor the code to enroll the new tools and methods to produce more readable and effective code. We will also mimic the development of real-life projects in the sense that at the start, we will have simple requirements, and later, new requirements will be set as our imagined business develops and sells more and more products. We will become imagined millionaires. We will use the code base of the previous chapter, and we will develop it further, though, in a new project. We will use Spring, Gradle, Tomcat, and soapUI, which are not new after we got acquainted with these in the previous chapter. In this chapter, you will learn the following topics: Annotation processing Using reflection Functional programming in Java using: Lambda expressions Streams Invoking scripts from Java The MyBusiness ordering The ordering process is a little bit more complicated than just looking up products. The order form itself lists products and amounts, and identifies who the customer for that order is. Identifiers give the products. All that we have to do is check that the products are available in our store, and we can deliver them to the given customer. This is the simplest approach; however, with some products, there are more restrictions. For example, when somebody orders a desk-side lamp, we deliver the power cord separately. The reason for this is that the power cord is specific to the country. We deliver different power cords to the United Kingdom and to Germany. One possible approach could be to identify the country of the customer. But this approach does not take into account that our customers are resellers. All customers could be located in the United Kingdom, and at the same time they may want to deliver the lamp with the power cable to Germany. To avoid such situations and ambiguity, it would be apt that our customers order the desk-side lamp and the power cord as separate items in the same order. In some cases, we deliver the desk-side lamp without the power cord, but this is a special case. We need some logic to identify these special cases. Therefore, we have to implement logic to see if there is a power cord for a desk-side lamp and if there is no automatic handling of the order, it is refused. It does not mean that we will not deliver the product. We will only put the order in a queue and some operator will have to look at it. The problem with this approach is that the desk-side lamp is only one product that needs configuration support. The more products we have, the more specialities they may have, and the piece of code that checks the consistency of an order becomes more and more complex until it reaches a level of complexity that is not manageable. When a class or method becomes too complex, the programmers refactor it, splitting up the method or class into smaller pieces. We have to do the same with the product checking. We shouldn't try to create one huge class that checks for the product and all the possible order constellations, but rather we should have many smaller checks so that each checks only one small set. Checking for consistency is simpler in some cases. Checking whether the lamp has a power cord has a complexity any novice programmer can program. We use this example in our code because we want to focus on the actual structure of the code, and not on the complex nature of the check itself. In real life, however, the checks can be fairly complex. Imagine a shop that sells computers. It puts a configuration together: power supply, graphic cards, and motherboard, the appropriate CPU, and the memory. There are many choices and some of them may not work together. In a real-life situation, we need to check that the motherboard is compatible with the memory selected, that it has as many banks as are in the order, that they are appropriately paired (some memories can only be installed in pairs), that there is a compatible slot for the graphics card, and that the power has enough watts to reliably run the whole configuration. This is very complex and is better not mixed up with the code that checks if there is a power cord for a lamp. Setting up the project Since we are still using Spring boot, the build file does not need any modification; we will use it as we will use the same file as in the last chapter. The package structure, however, is a bit different. This time, we do something more complicated than getting a request and responding to whatever the backend services deliver to us. Now, we have to implement complex business logic that, as we will see, needs many classes. When we have more than 10 classes, give or take, in a certain package, it is time to think about putting them into separate packages. The classes that are related to each other and have a similar functionality should be put into one package. This way, we will have a package for the following: The controllers (though we have only one in this example, but usually there are more) Data storing beans that have no more functionality than storing data, thus, fields, setters, and getters Checkers that will help us check power cords when a desk-side lamp is ordered Services that perform different services for the controller The main package for our program that contains the Application class, SpringConfiguration, and some interfaces Order controller and DTOs When a request comes to the server to order a bunch of products, it comes in an HTTPS POST request. The body of the request is encoded in JSON. Till now, we had controllers that were handling GET parameters, but handling POST requests is not much more difficult when we can rely on the data marshalling of Spring. The controller code itself is simple: package packt.java9.by.example.mybusiness.bulkorder.controllers; import ... @RestController public class OrderController { private Logger log = LoggerFactory.getLogger(OrderController.class); private final Checker checker; public OrderController(@Autowired Checker checker) { this.checker = checker; } @RequestMapping("/order") public Confirmation getProductInformation(@RequestBody Order order) { if (checker.isConsistent(order)) { return Confirmation.accepted(order); } else { return Confirmation.refused(order); } } } There is only one request that we handle in this controller: order. This is mapped to the URL, /order. The order is automatically converted from JSON to an order object from the request body. This is what the @RequestBody annotation asks Spring to do for us. The functionality of the controller simply checks the consistency of the order. If the order is consistent, then we accept the order; otherwise, we refuse it. The real-life example will also check that the order is not only consistent but also comes from a customer who is eligible for buying those products and that the products are available in the warehouse or, at least, can be delivered, based on the promises and lead time from the producers. To check the consistency of the order, we need something that does this job for us. As we know that we have to modularize the code and not implement too many things in a single class, we need a checker object. This is provided automatically based on the annotation on the class and also on the constructor of the controller by @Autowired. The Order class is a simple bean, simply listing the items: package packt.java9.by.example.mybusiness.bulkorder.dtos; import ...; public class Order { private String orderId; private List<OrderItem> items; private String customerId; ... setters and getters ... } The name of the package is dtos, which stands for the plural of Data Transfer Object (DTO). DTOs are objects that are used to transfer data between different components, usually over the network. Since the other side can be implemented in any language, the marshaling can be JSON, XML, or some other format that is capable of delivering nothing but data. These classes do not have real methods. DTOs usually have only fields, setters, and getters. The following is the class that contains one item in an order: package packt.java9.by.example.mybusiness.bulkorder.dtos; public class OrderItem { private double amount; private String unit; private String productId; ... setters and getters ... } The order confirmation is also in this package, and though this is also a true DTO, it has some simple auxiliary methods: package packt.java9.by.example.mybusiness.bulkorder.dtos; public class Confirmation { private final Order order; private final boolean accepted; private Confirmation(Order order, boolean accepted) { this.order = order; this.accepted = accepted; } public static Confirmation accepted(Order order) { return new Confirmation(order, true); } public static Confirmation refused(Order order) { return new Confirmation(order, false); } public Order getOrder() { return order; } public boolean isAccepted() { return accepted; } } We provide two factory methods for the class. This is a little violation of the single responsibility principle that purists hate. Most of the time, when the code becomes more complex, such short cuts bite back, and the code has to be refactored to be cleaner. The purist solution would be to create a separate factory class. The use of the factory methods either from this class or from a separated class makes the code of the controller more readable. The major task we have is the consistency check. The code, till this point, is almost trivial. Consistency checker We have a consistency checker class, and an instance of it is injected into the controller. This class is used to check the consistency, but it does not actually perform the check itself. It only controls the different checkers that we provide and invokes them one by one to do the real work. We require that a consistency checker, such as the one that checks whether the order contains a power cord when a desk-side lamp is ordered, implements the ConsistencyChecker interface: package packt.java9.by.example.mybusiness.bulkorder; import ... public interface ConsistencyChecker { boolean isInconsistent(Order order); } The method isInconsistent should return true if the order is inconsistent. It returns false if it does not know whether the order is inconsistent or not, but from the aspect that the actual checker examines the order, there is no inconsistency. Having several ConsistencyChecker classes, we have to invoke one after the other until one returns true or we are out of them. If none of them returns true, then we can safely assume, at least from the automated checkers' point of view, that the order is consistent. We know at the start of the development that we will really have a lot of consistency checkers and not all are relevant for all of the orders. We want to avoid the invocation of each checker for each order. To do so, we implement some filtering. We let products specify what type of checks they need. This is a piece of product information, such as the size or the description. To accommodate this, we need to extend the ProductInformation class. We will create each ConsistencyChecker interface, implementing the class to be a Spring bean (annotated with the @Component annotation), and at the same time, we will annotate them with an annotation that specifies what type of checks they implement. At the same time, ProductInformation is extended, containing a set of Annotation class objects that specify which checkers to invoke. We could simply list the checker classes instead of the annotations, but this gives us some extra freedom in configuring the mapping between the products and the annotations. The annotation specifies the type of the products, and the checker classes are annotated. The desk-side lamp has the PoweredDevice type, and the checker class, NeedPowercord, is annotated with the @PoweredDevice annotation. If there is any other type of products that also needs a power cord, then the annotation of that type should be added to the NeedPowercord class, and our code will work. Since we start diving deep into annotations and annotation handling, we have to first learn what annotations really are. We have already used annotations since Chapter 3, Optimizing the Sort, Making Code Professional but all we knew was how to use them, and that is usually dangerous without understanding what we did. Annotations Annotations are used with the @ character in front of them and can be attached to packages, classes, interfaces, fields, methods, method parameters, generic type declaration and use, and, finally, to annotations. Annotations can be used almost everywhere and they are used to describe some program meta information. For example, the @RestController annotation does not directly alter the behavior of the OrderController class. The behavior of the class is described by the Java code that is inside. The annotation helps Spring to understand what the class is and how it can and should be used. When Spring scans all the packages and classes to discover the different Spring beans, it sees the annotation on the class and takes it into account. There can be other annotations on the class that Spring does not understand. They may be used by some other framework or program code. Spring ignores them as any well-behaving framework. For example, as we will see later, we have in our code base, the NeedPowercord class , which is a Spring bean and, as such, annotated with the @Component annotation. At the same time, it is also annotated with the @PoweredDevice annotation. Spring has no idea about what a powered device is. This is something that we define and use. Spring ignores this. Packages, classes, interfaces, fields, and so on, can have many annotations attached to them. These annotations should simply be written in front of the declaration of the syntactical unit they are attached to. In the case of packages, the annotation has to be written in front of the package name in the package-info.java file. This file can be placed in the directory of the package and can be used to edit the JavaDoc for the package and also to add an annotation to the package. This file cannot contain any Java class since the name, package-info, is not a valid identifier. We cannot just write anything in front of anything as an annotation. Annotations should be declared. They are in the runtime of Java special interfaces. The Java file that declares the @PoweredDevice annotation, for example, looks like this: package packt.java9.by.example.mybusiness.bulkorder.checkers; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; @Retention(RetentionPolicy.RUNTIME) public @interface PoweredDevice { } The @ character in front of the interface keyword shows us that this is a special one: an annotation type. There are some special rules; for example, an annotation interface should not extend any other interface, not even an annotation one. On the other hand, the compiler automatically makes the annotation interface so that it extends the JDK interface, java.lang.annotation.Annotation. Annotations are in the source code, and thus, they are available during the compilation process. They can also be retained by the compiler and put into the generated class files, and when the class loader loads the class file, they may also be available during runtime. The default behavior is that the compiler stores the annotation along with the annotated element in the class file, but the class loader does not keep it available for runtime. To handle annotations during the compilation process, the Java compiler has to be extended using annotation processors. This is a fairly advanced topic and there are only a few examples you can meet while working with Java. An annotation processor is a Java class that implements a special interface and is invoked by the compiler when it processes an annotation in the source file that the processor is declared to have an interest in. Annotation retention Spring and other frameworks usually handle annotations during runtime. The compiler and the class loader have to be instructed that the annotation is to be kept available during runtime. To do so, the annotation interface itself has to be annotated using the @Retention annotation. This annotation has one parameter of the RetentionPolicy type, which is an enum. We will soon discuss how annotation parameters should be defined. It is interesting to note that the @Retention annotation on the annotation interface has to be available in the class file; otherwise, the class loaders would not know how to treat an annotation. How do we signal that an annotation is to be kept by the compiler after the compilation process? We annotate the annotation interface declaration. Thus, the declaration of @Retention is annotated by itself and it is declared to be available in runtime. The annotation declaration can be annotated using @Retention(RetentionPolicy.SOURCE), @Retention(RetentionPolicy.CLASS), or @Retention(RetentionPolicy.RUNTIME). Annotation target The last retention type will be the most frequent used. There are also other annotations that can be used on annotation declarations. The @Target annotation can be used to restrict the use of the annotation to certain locations. The argument to this annotation is either a single java.lang.annotation.ElementType value or an array of these values. There is a good reason to restrict the use of annotations. It is much better to get a compilation time error when we place an annotation in the wrong place than hunting during runtime why the framework ignores our annotation. Annotation parameters Annotations, as we saw, can have parameters. To declare these parameters in the @interface declaration of the annotation, we use methods. These methods have a name and a return value, but they should not have an argument. You may try to declare some parameters, but the Java compiler will be strict and will not compile your code. The values can be defined at the place where the annotation is used, using the name of the method and with the = character, assigning to them some value that is compatible with the type of the method. For example, let's suppose that we modify the declaration of the annotation PoweredDevice to the following: public @interface ParameteredPoweredDevice { String myParameter(); } In such a case, at the use of the annotation, we should specify some value for the parameter, such as the following: @Component @ParameteredPoweredDevice(myParameter = "1966") public class NeedPowercord implements ConsistencyChecker { ... If the name of the parameter is a value and at the place of use of the annotation there is no other parameter defined, then the name, "value", may be skipped. For example, modifying the code as follows is a handy shorthand when we have only one parameter: public @interface ParameteredPoweredDevice{ String value(); } ... @Component @ParameteredPoweredDevice("1966") public class NeedPowercord implements ConsistencyChecker { ... We can define optional parameters also using the default keyword following the method declaration. In this case, we have to define a default value for the parameter. Modifying the sample annotation we have further, we still can, but need not, specify the value. In the latter case, it will be an empty string: public @interface ParameteredPoweredDevice { String value() default ""; } Since the value we specify should be constant and calculable during compile time, there is not much use of complex types. Annotation parameters are usually strings, integers, and sometimes, doubles, or other primitive types. The exact list of the types given by the language specification is as follows: Primitive (double, int, and so on) String Class An enum Another annotation An array of any of the aforementioned types We have seen examples of String and also that enum:Retention and Target both have enum parameters. The interesting part we want to focus on is the last two items of the preceding list. When the value of the parameter is an array, the value can be specified as comma-separated values between the { and} characters. For example: String[] value(); This can then be added to the @interface annotation we can write: @ParameteredPoweredDevice({"1966","1967","1991"}) However, in case there is only one value we want to pass as the parameter value, we can still use the format: @ParameteredPoweredDevice("1966") In this case, the value of the attribute will be an array of length 1. When the value of an annotation is an array of annotation types, things get a bit more complex. We create an @interface annotation (note the plural in the name): @Retention(RetentionPolicy.RUNTIME) public @interface PoweredDevices { ParameteredPoweredDevice[] value() default {}; } The use of this annotation could be as follows: @PoweredDevices( {@ParameteredPoweredDevice("1956"), @ParameteredPoweredDevice({"1968", "2018"})} ) Note that this is not the same as having the ParameteredPoweredDevice annotation with three parameters. This is an annotation that has two parameters. Each parameter is an annotation. The first has one string parameter and the second has two. As you can see, annotations can be fairly complex, and some of the frameworks (or rather the programmers who created them) ran amok using them. Before you start writing a framework, research to see whether there is already a framework that you can use. Also, check whether there is some other way to solve your problem. 99% of annotation handling code could be avoided and made simpler. The less code we write for the same functionality, the happier we are. We programmers are the lazy types and this is the way it has to be. The last example, where the parameter of the annotation is an array of annotations, is important to understand how we can create repeatable annotations. Repeatable annotations Annotate the declaration of the annotation with @Repeatable to denote that the annotation can be applied multiple times at one place. The parameter to this annotation is an annotation type that should have a parameter of type, which is an array of this annotation. Don't try to understand! I'll give an example instead. I already have, in fact: we have @PoweredDevices. It has an argument that is an array of @ParameteredPoweredDevice. Consider that we now annotate this @interface as the following: ... @Repeatable(PoweredDevices.class) public @interface ParameteredPoweredDevice { ... Then, we can simplify the use of @ParameteredPoweredDevice. We can repeat the annotation multiple times and the Java runtime will automatically enclose it in the wrapping class, which, in this case, is @PoweredDevices. In this case, the following two will be equivalent: ... @ParameteredPoweredDevice("1956") @ParameteredPoweredDevice({"1968", "2018"}) public class NeedPowercord implements ConsistencyChecker { ... @PoweredDevices( {@ParameteredPoweredDevice("1956"), @ParameteredPoweredDevice({"1968", "2018"})} ) public class NeedPowercord implements ConsistencyChecker { ... The reason for this complex approach is again an example of backward compatibility that Java strictly follows. Annotations were introduced in Java 1.5 and repeatable annotations have been available only since version 1.8. We will soon talk about the reflection API that we use to handle the annotations during runtime. This API in the java.lang.reflect.AnnotatedElement interface has a getAnnotation(annotationClass) method, which returns an annotation. If a single annotation can appear more than once on a class, method, and so on, then there is no way of calling this method to get all the different instances with all the different parameters. Backward compatibility was ensured by introducing the containing type that wraps the multiple annotations. Annotation inheritance Annotations, just like methods or fields, can be inherited between class hierarchies. If an annotation declaration is marked with @Inherited, then a class that extends another class with this annotation can inherit it. The annotation can be overridden in case the child class has the annotation. Because there is no multiple inheritance in Java, annotations on interfaces cannot be inherited. Even when the annotation is inherited, the application code that retrieves the annotation of a certain element can distinguish between the annotations that are inherited and those that are declared on the entity itself. There are methods to get the annotations and separate methods to get the declared annotations that are declared on the actual element, and not inherited. @Documented annotations The @Documented annotation expresses the intent that the annotation is part of the contract of the entity and, this way, it has to get into the documentation. This is an annotation that the JavaDoc generator looks at when creating the documentation for an element that references the @Documented annotation. JDK annotations There are other annotations defined in the JDK in addition to those that are to be used to define annotations. We have already seen some of these. The most frequently used is the @Override annotation. When the compiler sees this annotation, it checks that the method really overrides some inherited method. Failing to do so will result in an error, saving us from miserable runtime debugging. The @Deprecated annotation signals in the documentation of a method, class, or some other element that the element is not to be used. It is still there in the code, because some users may still use it, but in the case of a new development that depends on the library containing the element, the newly developed code should not use it. The annotation has two parameters. One parameter is since, which can have a string value and may deliver version information about how long or since which version of the method, or class is deprecated. The other parameter is forRemoval, which should be true if the element will not appear in the future versions of the library. Some methods may be deprecated because there are better alternatives but the developers do not intend to remove the method from the library. In such a case, the forRemoval can be set to false. The @SuppressWarning annotation is also a frequently used one, though its use is questionable. It can be used to suppress some of the warnings of the compiler. It is recommended to write code, if possible, which can be compiled without any warning. The @FunctionalInterface annotation declares that an interface intends to have only one method. Such interfaces can be implemented as lambda expressions. You will learn about lambda expressions later in this chapter. When this annotation is applied on an interface and there is more than one method declared in the interface, the compiler will signal compilation error. This will prevent any developer early on from adding another method to an interface intended to be used together with functional programming and lambda expressions. Using reflection Now that you have learnt how to declare annotations and how to attach them to classes and methods, we can return to our ProductInformation class. Recall that we wanted to specify the type of products in this class and that each product type is represented by an @interface annotation. We have already listed it in the previous few pages, the one we will implement in our @PoweredDevice example. We will develop the code assuming that later there will be many such annotations, product types, and consistency checkers that are annotated with @Component and with one or more of our annotations. Getting annotations We will extend the ProductInformation class with the following field: private List<Class<? extends Annotation>> check; Since this is a DTO, and Spring needs the setters and getters, we will also add a new getter and setter to it. This field will contain the list of classes that each class implement one of our annotations and also the built-in JDK interface, Annotation, because that is the way the Java compiler generates them. At this point, this may be a bit murky but I promise that the dawn will break and there will be light as we go on. To get the product information, we have to look it up by ID. This is the interface and service that we developed in the last chapter, except, this time, we have another new field. This is, in fact, a significant difference although the ProductLookup interface did not change at all. In the last chapter, we developed two versions. One of the versions was reading the data from a properties file, the other one was connecting to a REST service. Properties files are ugly and old technology but a must if ever you intend to pass a Java interview or work on enterprise applications developed at the start of the 21st century. I had to include it in the last chapter. It was my own urge to include it in the book. At the same time, while coding for this chapter, I did not have the stomach to keep using it. I also wanted to show you that the same content could be managed in a JSON format. Now, we will extend the implementation of ResourceBasedProductLookup to read the product information from JSON formatted resource files. Most of the code remains the same in the class; therefore, we only list the difference here: package packt.java9.by.example.mybusiness.bulkorder.services; import ... @Service public class ResourceBasedProductLookup implements ProductLookup { private static final Logger log = LoggerFactory.getLogger(ResourceBasedProductLookup.class); private ProductInformation fromJSON(InputStream jsonStream) throws IOException { ObjectMapper mapper = new ObjectMapper(); return mapper.readValue(jsonStream, ProductInformation.class); } ... private void loadProducts() { if (productsAreNotLoaded) { try { Resource[] resources = new PathMatchingResourcePatternResolver(). getResources("classpath:products/*.json"); for (Resource resource : resources) { loadResource(resource); } productsAreNotLoaded = false; } catch (IOException ex) { log.error("Test resources can not be read", ex); } } } private void loadResource(Resource resource) throws IOException { final int dotPos = resource.getFilename().lastIndexOf('.'); final String id = resource.getFilename().substring(0, dotPos); final ProductInformation pi = fromJSON(resource.getInputStream()); pi.setId(id); products.put(id, pi); } ... In the project resources/products directory we have a few JSON files. One of them contains the desk lamp product information: { "id" : "124", "title": "Desk Lamp", "check": [ "packt.java9.by.example.mybusiness.bulkorder.checkers.PoweredDevice" ], "description": "this is a lamp that stands on my desk", "weight": "600", "size": [ "300", "20", "2" ] } The type of product is specified in a JSON array. In this example, this array has only one element and that element is the fully qualified name of the annotation interface that represents the type of product. When the JSON marshaller converts the JSON to a Java object, it recognizes that the field that needs this information is a List, so it converts the array to a list and, also, the elements from String to Class objects representing the annotation interface. Now that we have the resources loaded from JSON formatted resources and we saw how easy it is to read JSON data when using Spring, we can get back to the order consistency check. The Checker class implements the logic to collect the pluggable checkers and to invoke them. It also implements the annotation-based screening so as not to invoke the checkers we don't really need for the actual products in the actual order: package packt.java9.by.example.mybusiness.bulkorder.services; import ... @Component() @RequestScope public class Checker { private static final Logger log = LoggerFactory.getLogger(Checker.class); private final Collection<ConsistencyChecker> checkers; private final ProductInformationCollector piCollector; private final ProductsCheckerCollector pcCollector; public Checker( @Autowired Collection<ConsistencyChecker> checkers, @Autowired ProductInformationCollector piCollector, @Autowired ProductsCheckerCollector pcCollector) { this.checkers = checkers; this.piCollector = piCollector; this.pcCollector = pcCollector; } public boolean isConsistent(Order order) { Map<OrderItem, ProductInformation> map = piCollector.collectProductInformation(order); if (map == null) { return false; } Set<Class<? extends Annotation>> annotations = pcCollector.getProductAnnotations(order); for (ConsistencyChecker checker : checkers) { for (Annotation annotation : checker.getClass().getAnnotations()) { if (annotations.contains( annotation.annotationType())) { if (checker.isInconsistent(order)) { return false; } break; } } } return true; } } One of the interesting things to mention is that the Spring auto-wiring is very clever. We have a field with the Collection<ConsistencyChecker> type. Usually, auto-wiring works if there is exactly one class that has the same type as the resources to wire. In our case, we do not have any such candidate since this is a collection, but we have many ConsistencyChecker classes. All our checkers implement this interface and Spring recognizes it, instantiates them all, magically creates a collection of them, and injects the collection into this field. Usually a good framework works logically. I was not aware of this feature of Spring, but I thought that this would be logical and, magically, it worked. If things are logical and just work, you do not need to read and remember the documentation. A bit of caution does not harm however. After I experienced that this functionality works this way, I looked it up in the documentation to see that this is really a guaranteed feature of Spring and not something that just happens to work but may change in future versions without notice. Using only guaranteed features is extremely important but is neglected many times in our industry. When the isConsistent method is invoked, it first collects the product information into HashMap, assigning a ProductInformation instance to each OrderItem. This is done in a separate class. After this, ProductsCheckerCollector collects the ConsistencyChecker instances needed by one or more product items. When we have this set, we need to invoke only those checkers that are annotated with one of the annotations that are in this set. We do that in a loop. In this code, we use reflection. We loop over the annotations that each checker has. To get the collection of annotations, we invoke checker.getClass().getAnnotations(). This invocation returns a collection of objects. Each object is an instance of some JDK runtime generated class that implements the interface we declared as an annotation in its own source file. There is no guarantee, though, that the dynamically created class implements only our @interface and not some other interfaces. Therefore, to get the actual annotation class, we have to invoke the annotationType method. The ProductCheckerCollector and ProductInformationCollector classes are very simple, and we will discuss them later when we learn about streams. They will serve as a good example at that place, when we implement them using loops and, right after that, using streams. Having them all, we can finally create our actual checker classes. The one that helps us see that there is a power cord ordered for our lamp is the following: package packt.java9.by.example.mybusiness.bulkorder.checkers; import ... @Component @PoweredDevice public class NeedPowercord implements ConsistencyChecker { private static final Logger log = LoggerFactory.getLogger(NeedPowercord.class); @Override public boolean isInconsistent(Order order) { log.info("checking order {}", order); CheckHelper helper = new CheckHelper(order); return !helper.containsOneOf("126", "127", "128"); } } The helper class contains simple methods that will be needed by many of the checkers, for example: public boolean containsOneOf(String... ids) { for (final OrderItem item : order.getItems()) { for (final String id : ids) { if (item.getProductId().equals(id)) { return true; } } } return false; } Invoking methods In this example, we used only one single reflection call to get the annotations attached to a class. Reflection can do many more things. Handling annotations is the most important use for these calls since annotations do not have their own functionality and cannot be handled in any other way during runtime. Reflection, however, does not stop telling us what annotations a class or any other annotable element has. Reflection can be used to get a list of the methods of a class, the name of the methods as strings, the implemented interfaces of a class, the parent class it extends, the fields, the types of fields, and so on. Reflection generally provides methods and classes to walk through the actual code structure down to the method level, programmatically. This walkthrough does not only allow reading types and code structure but also makes it possible to set field values and call methods without knowing the methods' name at compile time. We can even set fields that are private and are not generally accessible by the outside world. It is also to note that accessing the methods and fields through reflection is usually slower than through compiled code because it always involves lookup by the name of the element in the code. The rule of thumb is that if you see that you have to create code using reflection, then realize that you are probably creating a framework (or writing a book about Java that details reflection). Does it sound familiar? Spring also uses reflection to discover the classes, methods, and fields, and also to inject an object. It uses the URL class loader to list all the JAR files and directories that are on the class path, loads them, and examines the classes. For a contrived example, for the sake of demonstration, let's assume that the ConsistencyChecker implementations were written by many external software vendors, and the architect who originally designed the program structure just forgot to include the isConsistent method in the interface. (At the same time, to save our mental health, we can also imagine that this person is not working anymore in the company for doing so.) As a consequence, the different vendors delivered Java classes that "implement" this interface but we cannot invoke the method, not only because we do not have a common parent interface that has this method but also because the vendors just happened to use different names for their methods. What can we do in this situation? Business-wise, asking all the vendors to rewrite their checkers is ruled out because them knowing we are in trouble attaches a hefty price tag to the task. Our managers want to avoid that cost and we developers also want to show that we can mend the situation and do miracles. (Later, I will have a comment on that.) We could just have a class that knows every checker and how to invoke each of them in many different ways. This would require us to maintain the said class whenever a new checker is introduced to the system, and we want to avoid that. The whole plugin architecture we are using was invented for this very purpose in the first place. How can we invoke a method on an object that we know has only one declared method, which accepts an order as a parameter? That is where reflection comes into the picture. Instead of calling checker.isInconsistent(order), we implement a small private method, isInconsistent, which calls the method, whatever its name is, via reflection: private boolean isInconsistent(ConsistencyChecker checker, Order order) { Method[] methods = checker.getClass().getDeclaredMethods(); if (methods.length != 1) { log.error( "The checker {} has zero or more than one methods", checker.getClass()); return false; } final Method method = methods[0]; final boolean inconsistent; try { inconsistent = (boolean) method.invoke(checker, order); } catch (InvocationTargetException | IllegalAccessException | ClassCastException e) { log.error("Calling the method {} on class {} threw exception", method, checker.getClass()); log.error("The exception is ", e); return false; } return inconsistent; } We can get the class of the object by calling the getClass method, and on the object that represents the class itself, we can call getDeclaredMethods. Fortunately, the checker classes are not littered by many methods, so we check that there is only one method declared in the checker class. Note that there is also a getMethods method in the reflection library but it always will return more than one method. It returns the declared and the inherited methods. Because each and every class inherits from java.lang.Object, at least the methods of the Object class will be there. After this, we try to invoke the class using the Method object that represents the method in the reflection class. Note that this Method object is not directly attached to an instance. We retrieved the method from the class, and thus, when we invoke it, we should pass the object it should work on as a first parameter. This way, x.y(z), becomes method.invoke(x,z). The last parameter of invoke is a variable number of arguments that are passed as an Object array. In most cases, when we invoke a method, we know the arguments in our code even if we do not know the name of the method and have to use reflection. When even the arguments are not known but are available as a matter of calculation, then we have to pass them as an Object array. Invoking a method via reflection is a risky call. If we try to call a method the normal way, which is private, then the compiler will signal an error. If the number of arguments or types are not appropriate, the compiler will again will give us an error. If the returned value is not boolean, or there is no return value at all, then we again get a compiler error. In the case of reflection, the compiler is clueless. It does not know what method we will invoke when the code is executing. The invoke method, on the other hand, can and will notice all these failures when it is invoked. If any of the aforementioned problems occur, then we will get exceptions. If the invoke method itself sees that it cannot perform what we ask of it, then it will throw InvocationTargetException or IllegalAccessException. If the conversion from the actual return value to boolean is not possible, then we will get ClassCastException. About doing magic, it is a natural urge that we feel like making something extraordinary, something outstanding. This is okay when we are experimenting with something, doing a hobby job. On the other hand, this is strongly not okay when we are working on a professional job. Average programmers, who do not understand your brilliant solution, will maintain the code in an enterprise environment. They will turn your nicely combed code into haystack while fixing some bugs or implementing some minor new features. Even if you are the Mozart of programming, they will be, at best, no-name singers. A brilliant code in an enterprise environment can be a requiem, with all the implications of that metaphor. Last but not least, the sad reality is that we are usually not the Mozarts of programming. Note that in case the return value of the original value is primitive, then it will be converted to an object by reflection, and then we will convert it back to the primitive value. If the method does not have a return value, in other words, if it is void, then the reflection will return a java.lang.Void object. The Void object is only a placeholder. We cannot convert it to any primitive value or any other type of objects. It is needed because Java is strict and invoke has to return an Object, so the runtime needs something that it can return. All we can do is check that the returned value class is really Void. Let's go on with the storyline and our solution. We submitted the code and it works in production for a while till a new update from a software vendor breaks it. We debug the code in the test environment and see that the class now contains more than one method. Our documentation clearly states that they should only have one public method, and they provided a code that has...hmm...we realize that the other methods are private. They are right; they can have private methods according to the contract, so we have to amend the code. We replace the lines that look up the one and only method: Method[] methods = checker.getClass().getDeclaredMethods(); if (methods.length != 1) { ... } final Method method = methods[0]; The new code will be as follows: final Method method = getSingleDeclaredPublicMethod(checker); if (method == null) { log.error( "The checker {} has zero or more than one methods", checker.getClass()); return false; } The new method we write to look up the one and only public method is as follows: private Method getSingleDeclaredPublicMethod( ConsistencyChecker checker) { final Method[] methods = checker.getClass().getDeclaredMethods(); Method singleMethod = null; for (Method method : methods) { if (Modifier.isPublic(method.getModifiers())) { if (singleMethod != null) { return null; } singleMethod = method; } } return singleMethod; } To check whether the method is public or not, we use a static method from the Modifier class. There are methods to check all possible modifiers. The value that the getModifiers method returns is an int bit field. Different bits have different modifiers and there are constants that define these. This simplification leads to inconsistency, which you can check if a method is an interface or volatile, that is, actually nonsense. The fact is that bits that can only be used for other types of reflection objects will never be set. There is one exception, which is volatile. This bit is reused to signal bridge methods. Bridge methods are created by the compiler automatically and can have deep and complex issues that we do not discuss in this book. The reuse of the same bit does not cause confusion because a field can be volatile, but as a field, it cannot be a bridge method. Obviously, a field is a field and not a method. In the same way, a method cannot be a volatile field. The general rule is: do not use methods on reflection objects where they do not have a meaning; or else, know what you do. Making the storyline even more intricate, a new version of a checker accidentally implements the checking method as a package private. The programmer simply forgot to use the public keyword. For the sake of simplicity, let's assume that the classes declare only one method again, but it is not public. How do we solve this problem using reflection? Obviously, the simplest solution is to ask the vendors to fix the problem: it is their fault. In some cases, however, we must create a workaround over some problems. There is another solution: creating a class with a public method in the same package, invoking the package private methods from the other class, thus relaying the other class. As a matter of fact, this solution, as a workaround for such a bug, seems to be more logical and cleaner, but this time, we want to use reflection. To avoid java.lang.IllegalAccessException, we have to set the method object as accessible. To do so, we have to insert the following line in front of the invocation: method.setAccessible(true); Note that this will not change the method to public. It will only make the method accessible for invocation through the very instance of the method object that we set as accessible. I have seen code that checks whether a method is accessible or not by calling the isAccessible method and saves this information; it sets the method as accessible if it was not accessible and restores the original accessibility after the invocation. This is totally useless. As soon as the method variable goes out of scope, and there is no reference to the object we set the accessibility flag to, the effect of the setting wears off. Also, there is no penalty for setting the accessibility of a public or an otherwise callable method. Setting fields We can also call setAccessible on Field objects and then we can even set the value of private fields using reflection. Without further fake stories, just for the sake of the example, let's make a ConsistencyChecker named SettableChecker: @Component @PoweredDevice public class SettableChecker implements ConsistencyChecker { private static final Logger log = LoggerFactory.getLogger(SettableChecker.class); private boolean setValue = false; public boolean isInconsistent(Order order) { return setValue; } } This checker will return false, unless we set the field to true using reflection. We do set it as such. We create a method in the Checker class and invoke it from the checking process for each checker: private void setValueInChecker(ConsistencyChecker checker) { Field[] fields = checker.getClass().getDeclaredFields(); for( final Field field : fields ){ if( field.getName().equals("setValue") && field.getType().equals(boolean.class)){ field.setAccessible(true); try { log.info("Setting field to true"); field.set(checker,true); } catch (IllegalAccessException e) { log.error("SNAFU",e); } } } } The method goes through all the declared fields and if the name is setValue and the type is boolean, then it sets it to true. This will essentially render all orders that contain a powered device as rejected. Note that although boolean is a built-in language primitive, which is not a class by any means, it still has a class so that reflection can compare the type of the field gainst he class that boolean artificially has. Now boolean.class is a class literal in the language, and for each primitive, a similar constant can be used. The compiler identifies these as class literals and creates the appropriate pseudo class references in the byte code so that primitives can also be checked in this way, as demonstrated in the sample code of the setValueInChecker method. We checked that the field has the appropriate type, and we also called the setAccessible method on the field. Even though the compiler does not know that we really did everything to avoid IllegalAccessException, it still believes that calling set on field can throw such an exception, as it is declared. However, we know that it should not happen. (Famous last words of a programmer?) To handle this situation, we surround the method call with a try block, and in the catch branch, we log the exception. Functional programming in Java Since we have created a lot of code in our example for this chapter, we will look at the functional programming features of Java, which will help us delete many lines from our code. The less code we have, the easier it is to maintain the application; thus, programmers love functional programming. But this is not the only reason why functional programming is so popular. It is also an excellent way to describe certain algorithms in a more readable and less error prone manner than conventional loops. Functional programming is not a new thing. The mathematical background was developed for it in the 1930s. One of the first (if not the first) functional programming languages is LISP. It was developed in the 1950s and it is still in use, so much that there is a version of the language implemented on the JVM (Clojure). Functional programming, in short, means that we express the program structure in terms of functions. In this meaning, we should think of functions as in mathematics and not as the term is used in programming languages such as C. In Java, we have methods, and when we are following the functional programming paradigm, we create and use methods that behave like mathematical functions. A method is functional if it gives the same result no matter how many times we invoke it, just as sin(0) is always zero. Functional programming avoids changing the state of objects, and because the state is not changing, the results are always the same. This also eases debugging. If a function has once returned a certain value for the given arguments, it will always return the same value. We can also read the code as a declaration of the calculation more than as commands that are executed one after the other. If the execution order is not important, then the readability of the code may also increase. Java helps functional programming style with lambda expressions and streams. Note that these streams are not I/O streams and do not really have any relation to those. We will first take a short look at lambda expressions and what streams are, and then, we will convert some parts of our program to use these programming constructs. We will also see how much more readable these codes become. Readability is a debatable topic. A code may be readable to one developer and may be less readable to another. It very much depends on what they got used to. I experience many times that developers get distracted with streams. When developers first meet streams, the way to think about them and how they look is just strange. But this is the same as starting to learn using a bicycle. While you are still learning its use and you fall more than you roll, it is definitely slower than walking. On the other hand, once you have learnt how to ride a bike... Lambda We have already used lambda expressions in Chapter 3, Optimizing the Sort - Making Code Professional when we wrote the exception-throwing test. In that code, we set the comparator to a special value that was throwing RuntimeException at each invocation: sort.setComparator((String a, String b) -> { throw new RuntimeException(); }); The argument type is Comparator; therefore, what we have to set there should be an instance of a class that implements the java.util.Comparator interface. That interface defines only one method that implementations have to define: compare. Thus, we can define it as a lambda expression. Without lambda, if we need an instance, we have to type a lot. We have to create a class, name it, declare the compare method in it, and write the body of the method, as shown in the following code segment: public class ExceptionThrowingComparator implements Comparator { public int compare(T o1, T o2){ throw new RuntimeException(); } } At the location of use, we should instantiate the class and pass it as an argument: sort.setComparator(new ExceptionThrowingComparator()); We may save a few characters if we define the class as an anonymous class but the overhead is still there. What we really need is the body of the one and single method that we have to define. This is where lambda comes into the picture. We can use a lambda expression in any place where we would otherwise need an instance of a class that has to define only one method. The methods that are defined and inherited from Object do not count, and we also do not care about the methods that are defined as default methods in the interface. They are there. Lambda defines the one that is not yet defined. In other words, lambda clearly depicts, with much less overhead as an anonymous class, that the value is a functionality that we pass as a parameter. The simple form of a lambda expression is as follows: parameters -> body The parameters can be enclosed between parentheses or can only stand without. The body similarly can be enclosed between the { and} characters or it can be a simple expression. This way a lambda expression can reduce the overhead to a minimum, using the parentheses only where they are really needed. It is also an extremely useful feature of lambda expressions that we do not need to specify the types of the parameters in case it is obvious from the context where we use the expression. Thus, the preceding code segment can even be shorter, as follows: sort.setComparator((a, b) -> { throw new RuntimeException(); }); The parameters, a and b, will have the type as needed. To make it even simpler, we can also omit the ( and) characters around the parameters in case there is only one. The parentheses are not optional if there is more than one parameter. This is to avoid ambiguity in some situations. For example, the method call, f(x,y->x+y) could have been a method with two arguments: x and a lambda expression that has one parameter, y. At the same time, it could also be a method call with a lambda expression that has two parameters, x and y. Lambda expressions are very handy when we want to pass functionality as an argument. The declaration of the type of argument at the place of the method declaration should be a functional interface type. These interfaces can optionally be annotated using @FunctionalInterface. The Java runtime has many such interfaces defined in the java.util.function package. We will discuss some of them in the next section along with their use in streams. For the rest, the standard Java documentation is available from Oracle. Streams Streams were also new in Java 8, just like lambda expressions. They work together very strongly, so their appearance at the same time is not a surprise. Lambda expressions as well as streams support the functional programming style. The very first thing to clarify is that streams do not have anything to do with input and output streams, except the name. They are totally different things. Streams are more like collections with some significant differences. (If there were no differences, they would just have been collections.) Streams are essentially pipelines of operations that can run sequentially or in parallel. They obtain their data from collections or other sources, including data that is manufactured on-the-fly. Streams support the execution of the same calculation on multiple data. This structure is referred to as Single Instruction Multiple Data (SIMD). Don't be afraid of the expression. This is a very simple thing. We have already done that many times in this book. Loops are also kind of SIMD structures. When we loop through the checker classes to see whether any of those opposes the order, we perform the same instruction for each and every checker. Multiple checkers are multiple data. One problem with loops is that we define the order of execution when it is not needed. In the case of checkers, we do not really care what order the checkers are executed in. All we care about is that all are okay with the order. We still specify some order when we program the loop. This comes from the nature of loops, and there is no way we could change that. That is how they work. However, it would be nice if we could just, somehow, say "do this and that for each and every checker". This is one point where streams come into the picture. Another point is that code that uses loops is more imperative rather than descriptive. When we read the program of a loop construct, we focus on the individual steps. We first see what the commands in the loop do. These commands work on the individual elements of the data and not on the whole collection or array. Later putting the individual steps together in our brain we realize what the big picture is, what the loop is for. In the case of streams, the description of operations is a level higher. Once we learn the stream methods, it is easier to read them. Stream methods work on the whole stream and not on the individual elements, and thus are more descriptive. is an interface. An object with a type implementing this interface represents many objects and provides methods that can be used to perform instructions on these objects. The objects may or may not be available when we start the operation on one of them, or may just be created when needed. This is up to the actual implementation of the Stream interface. For example, suppose we generate a stream that contains int values using the following code: java.lang.Stream IntStream.iterate( 0, (s) -> s+1 ) In the preceding code snippet, all the elements cannot be generated because the stream contains an infinite number of elements. This example will return the numbers 0, 1, 2, and so on until further stream operations, which are not listed here, terminate the calculation. When we program Stream, we usually create a stream from a Collection—not always, but many times. The Collection interface was extended in Java 8 to provide the stream and parallelStream methods. Both of them return stream objects that represent the elements of the collection. While stream returns the elements in the same order as they are in the collection in case there is a natural order, the parallelStream creates a stream that may be worked on in a parallel manner. In this case, if some of the methods that we use on the stream are implemented in that way, the code can use the multiple processors available in the computer. As soon as we have a stream, we can use the methods that the Stream interface defines. The one to start with is forEach. This method has one argument, which is usually provided as a lambda expression and will execute the lambda expression for each element of the stream. In the Checker class, we have the isConsistent method. In this method, there is a loop that goes through the annotations of the checker class. If we wanted to log the interfaces that the annotation in the loop implements, we could add the following: for (ConsistencyChecker checker :checkers) { for (Annotation annotation : checker.getClass().getAnnotations()) { Arrays.stream(annotation.getClass().getInterfaces()) .forEach( t ->log.info("annotation implemented interfaces {}",t) ); ... In this example, we create a stream from an array using the factory method from the Arrays class. The array contains the interfaces returned by the reflection method, getInterfaces. The lambda expression has only one parameter; thus, we do not need to use parentheses around it. The body of the expression is a method call that returns no value; thus, we also omit the { and } characters. Why all this hassle? What is the gain? Why couldn't we just write a simple loop that logs the elements of the array? The gains are readability and maintainability. When we create a program, we have to focus on what the program should do and not on how it should do it. In an ideal world, a specification would just be executable. We may actually get there in the future when programming work will be replaced by artificial intelligence. (Not the programmers, though.) We are not there, yet. We have to tell the computers how to do what we want to achieve. We used to have to enter binary codes on the console of PDP-11 to get machine code deployed into the memory to have it executed. Later, we had assemblers; still later, we had FORTRAN and other high-level programming languages that have replaced much of the programming work as it was 40 years ago. All these developments in programming shift the direction from how towards what. Today, we program in Java 9, and the road still has miles to go. The more we can express what to do instead of how to do, the shorter and more understandable our programs will be. It will contain the essence and not some artificial litter that is needed by the machines to just do what we want. When I see a loop in a code I have to maintain, I assume that there is some importance of the order in which the loop is executed. There may be no importance at all. It may be obvious after a few seconds. It may need minutes or more to realize that the ordering is not important. This time is wasted and can be saved with programming constructs that better express the what to do part instead of the how to do. Functional interfaces The argument to the method should be java.util.function.Consumer. This is an interface that requires the accept method to be defined, and this method is void. The lambda expression or a class that implements this interface will consume the argument of the accept method and does not produce anything. There are several other interfaces defined in that package, each serving as a functional interface used to describe some method arguments that can be given as lambda expressions in the actual parameters. For example, the opposite of Consumer is Supplier. This interface has a method named get that does not need any argument but gives some Object as a return value. If there is an argument and also a returned value, the interface is called Function. If the returned value has to be the same type as the argument, then the UnaryOperator interface is our friend. Similarly, there is a BinaryOperator interface, which returns an object of the same type as the arguments. Just as we got from Function to UnaryOperator, we can see that in the other direction, there is also BiFunction in case the arguments and the return values do not share the type. These interfaces are not defined independently of each other. If a method requires Function and we have UnaryOperator to pass, it should not be a problem. UnaryOperator is nothing else but Function that has the same type of arguments. A method that can work with Function, which accepts an object and returns an object, should not have a problem if they have the same type. Those can be, but need not be, different. To let that happen, the UnaryOperator interface extends Function and thus can be used in the place of Function. The interfaces in this class we met so far are defined using generics. Because generic types cannot be primitives, the interfaces that operate on primitive values should be defined separately. Predicate, for example, is an interface that defines booleantest(T t). It is a function that returns a boolean value and is used many times in stream methods. There are also interfaces, such as BooleanSupplier, DoubleConsumer, DoubleToIntFunction, and more, that work with primitive boolean, double, and int. The number of possible combinations of the different argument types and return values is infinite... almost. Fun fact: To be very precise, it is not infinite. A method can have at most 254 arguments. This limit is specified in the JVM and not in the Java language specification. Of course, one is useless without the other. There are 8 primitive types (plus Object, plus the possibility that there are less than 254 arguments), which means that the total number of possible functional interfaces is 10254, give or take, a few magnitudes. Practically, infinite! We should not expect to have all the possible interfaces defined in the JDK in this package. These are only those interfaces that are the most useful. There is no interface, for example, that uses short or char. If we need anything like that, then we can define the interface in our code. Or just think hard and find out how to use an already defined one. (I have never used the short type during my professional carrier. It was never needed.) How are these functional interfaces used in streams? The Stream interface defines the methods that have some functional interface types as arguments. For example, the allMatch method has a Predicate argument and returns a Boolean value, which is true if all the elements in the stream match Predicate. In other words, this method returns true if and only if Predicate, supplied as an argument, returns true for each and every element of the stream. In the following code, we will rewrite some of the methods that we implemented in our sample code using loops to use streams, and through these examples, we will discuss the most important methods that streams provide. We saved up two classes, ProductsCheckerCollector and ProductInformationCollector, to demonstrate the stream usage. We can start with these. ProductsCheckerCollector goes through all the products that are contained in the Order and collects the annotations that are listed in the products. Each product may contain zero, one, or many annotations. These are available in a list. The same annotation may be referenced multiple times. To avoid duplicates, we use HashSet, which will contain only one instance of the elements even if there are multiple instances in the products: public class ProductsCheckerCollector { private final ProductInformationCollector pic; public ProductsCheckerCollector(@Autowired ProductInformationCollector pic) { this.pic = pic; } public Set<Class<? extends Annotation>> getProductAnnotations(Order order) { Map<OrderItem, ProductInformation> piMap = pic.collectProductInformation(order); final Set<Class<? extends Annotation>> annotations = new HashSet<>(); for (OrderItem item : order.getItems()) { final ProductInformation pi = piMap.get(item); if (pi != null && pi.getCheck() != null) { for (Class<? extends Annotation> check : pi.getCheck()) { annotations.addAll(pi.getCheck()); } } return annotations; } } Now, let's see how this method looks when we recode it using streams: public Set<Class<? extends Annotation>> getProductAnnotations(Order order) { Map<OrderItem, ProductInformation> piMap = pic.collectProductInformation(order); return order.getItems().stream() .map(piMap::get) .filter(Objects::nonNull) .peek(pi -> { if (pi.getCheck() == null) { log.info("Product {} has no annotation", pi.getId()); } }) .filter(pi -> pi.getCheck() != null) .peek(pi -> log.info("Product {} is annotated with class {}", pi.getId(), pi.getCheck())) .flatMap(pi -> pi.getCheck().stream()) .collect(Collectors.toSet()); } The major work of the method gets into a single, though huge, stream expression. We will cover the elements of the expression in the coming pages. List returned by order.getItems is converted calling the stream method: returnorder.getItems().stream() As we have already mentioned it briefly, the stream method is part of the Collection interface. Any class that implements the Collection interface will have this method, even those that were implemented before streams were introduced in Java 8. This is because the stream method is implemented in the interface as a default method. This way, if we happen to implement a class implementing this interface, even if we do not need streams, we get it for free as an extra. The default methods in Java 8 were introduced to support backward compatibility of interfaces. Some of the interfaces of the JDK were to be modified to support lambda and functional programming. One example is the stream method. With the pre-Java 8 feature set, the classes implementing some of the modified interfaces should have been modified. They would have been required to implement the new method. Such a change is not backward compatible, and Java as a language and JDK was paying keen attention to be backward compatible. Thus, default methods were introduced. These let a developer extend an interface and still keep it backward compatible, providing a default implementation for the methods, which are new. Contrary to this philosophy, brand new functional interfaces of Java 8 JDK also have default methods, though, having no prior version in the JDK, they have nothing to be compatible with. In Java 9, interfaces were also extended and now they can contain not only default and static methods but also private methods. This way, interfaces became kind of equivalent to abstract classes, though there are no fields in an interface except constant static fields. This interface functionality open up is a much criticized feature, which just poses the programming style and structural issues that other languages allowing multiple class inheritance face. Java was avoiding this till Java 8 and Java 9. What is the take-away from this? Be careful with default methods and also with private methods in interfaces. Use them wisely if at all. The elements of this stream are OrderItem objects. We need ProductInformation for each OrderItem. Method references Lucky that we have Map, which pairs order items with product information, so we can invoke get on Map: .map(piMap::get) The map method is again something that has the same name as something else in Java and should not be confused. While the Map class is a data structure, the map method in the Stream interface performs mapping of the stream elements. The argument of the method is a Function (recall that this is a functional interface we recently discussed). This function converts a value, T, which is available as the element of the original stream (Stream<T>) to a value, R, and the return value of the map method is Stream<R>. The map method converts Stream<T> to Stream<R> using the given Function<T,R>, calling it for each element of the original stream and creating a new stream from the converted elements. We can say that the Map interface maps keys to values in a data structure in a static way, and the Stream method, map, maps one type of values to another (or the same) type of values dynamically. We have already seen that we can provide an instance of a functional interface in the form of a lambda expression. This argument is not a lambda expression. This is a method reference. It says that the map method should invoke the get method on Map piMap using the actual stream element as an argument. We are lucky that get also needs one argument, aren't we? We could also write as follows: .map( orderItem ->piMap.get(orderItem)) However, this would have been exactly the same as piMap::get. This way, we can reference an instance method that works on a certain instance. In our example, the instance is the one referenced by the piMap variable. It is also possible to reference static methods. In this case, the name of the class should be written in front of the :: characters. We will soon see an example of this when we will use the static method, nonNull, from the Objects class (note that the class name is in plural, and it is in the java.util package and not java.lang). It is also possible to reference an instance method without giving the reference on which it should be invoked. This can be used in places where the functional interface method has an extra first parameter, which will be used as the instance. We have already used this in Chapter 3, Optimizing the Sort - Making Code Professional, when we passed String::compareTo, when the expected argument was a Comparator. The compareTo method expects one argument, but the compare method in the Comparator interface needs two. In such a situation, the first argument will be used as the instance on which compare has to be invoked and the second argument is passed to compare. In this case, String::compareTo is the same as writing the lambda expression (String a, String b) -> a.compareTo(b). Last but not least, we can use method references to constructors. When we need a Supplier of (let's be simple) Object, we can write Object::new. The next step is to filter out the null elements from the stream. Note that, at this point, the stream has ProductInformation elements: .filter(Objects::nonNull) The filter method uses Predicate and creates a stream that contains only the elements that match the predicate. In this case, we used the reference to a static method. The filter method does not change the type of stream. It only filters out the elements. The next method we apply is a bit anti-functional. Pure functional stream methods do not alter the state of any object. They create new objects that they return but, other than that, there is no side effect. peek itself is no different because it only returns a stream of the same elements as the one it is applied on. However, this no-operation feature lures the novice programmer to do something non-functional and write code with side-effects. After all, why use it if there is no (side) effect in calling it? .peek(pi -> { if (pi.getCheck() == null) { log.info("Product {} has no annotation", pi.getId()); } }) While the peek method itself does not have any side effects, the execution of the lambda expression may have. However, this is also true for any of the other methods. It is just the fact that, in this case, it is more tempting to do something inadequate. Don't. We are disciplined adults. As the name of the method suggests, we may peek into the stream but we are not supposed to do anything else. With programming being a particular activity, in this case, peeking, is adequate. And that is what we actually do in our code: we log something. After this, we get rid of the elements that have no ProductInformation; we also want to get rid of the elements that have, but there is no checker defined: .filter(pi ->pi.getCheck() != null) In this case, we cannot use method references. Instead, we use a lambda expression. As an alternative solution, we may create a boolean hasCheck method in ProductInformation, which returns true if the private field check is not null. This would then read as follows: .filter(ProductInformation::hasCheck) This is totally valid and works, although the class does not implement any functional interface and has many methods, not only this one. However, the method reference is explicit and specifies which method to invoke. After this second filter, we log the elements again: .peek(pi -> log.info( "Product {} is annotated with class {}", pi.getId(), pi.getCheck())) The next method is flatMap and this is something special and not easy to comprehend. At least for me, it was a bit more difficult than understanding map and filter when I learned functional programming: .flatMap(pi ->pi.getCheck().stream()) This method expects that the lambda, method reference, or whatever is passed to it as an argument, creates a whole new stream of objects for each element of the original stream the method is invoked on. The result is, however, not a stream of streams, which also could be possible, but rather the returned streams are concatenated into one huge stream. If the stream we apply it to is a stream of integer numbers, such as 1, 2, 3, ..., and the function for each number n returns a stream of three elements n, n+1, and n+2, then the resulting stream, flatMap, produces a stream containing 1, 2, 3, 2, 3, 4, 3, 4, 5, 4, 5, 6, and so on. Finally, the stream we have should be collected to a Set. This is done by calling the collector method: .collect(Collectors.toSet()); The argument to the collector method is (again a name overuse) Collector. It can be used to collect the elements of the stream into some collection. Note that Collector is not a functional interface. You cannot just collect something using a lambda or a simple method. To collect the elements, we definitely need some place where the elements are collected as the ever-newer elements come from the stream. The Collector interface is not simple. Fortunately, the java.util.streams.Collectors class (again note the plural) has a lot of static methods that create and return Object that create and return Collector objects. One of these is toSet, which returns a Collector that helps collect the elements of the stream into a Set. The collect method will return the Set when all the elements are there. There are other methods that help collect the stream elements by summing up the elements, calculating the average, or to a List, Collection, or to a Map. Collecting elements to a Map is a special thing, since each element of a Map is actually a key-value pair. We will see the example for that when we look at ProductInformationCollector. The ProductInformationCollector class code contains the collectProductInformation method, which we will use from the Checker class as well as from the ProductsCheckerCollector class: private Map<OrderItem, ProductInformation> map = null; public Map<OrderItem, ProductInformation> collectProductInformation(Order order) { if (map == null) { map = new HashMap<>(); for (OrderItem item : order.getItems()) { final ProductInformation pi = lookup.byId(item.getProductId()); if (!pi.isValid()) { map = null; return null; } map.put(item, pi); } } return map; } The simple trick is to store the collected value in Map, and if that is not null, then just return the already calculated value, which may save a lot of service calls in case this method is called more than once handling the same HTTP request. There are two ways of coding such a structure. One is checking the non-nullity of the Map and returning if the Map is already there. This pattern is widely used and has a name. This is called guarding if. In this case, there is more than one return statement in the method, which may be seen as a weakness or anti-pattern. On the other hand, the tabulation of the method is one tab shallower. It is a matter of taste and in case you find yourself in the middle of a debate about one or the other solution, just do yourself a favor and let your peer win on this topic and save your stamina for more important issues, for example, whether you should use streams or just plain old loops. Now, let's see how we can convert this solution into a functional style: public Map<OrderItem, ProductInformation> collectProductInformation(Order order) { if (map == null) { map = order.getItems() .stream() .map(item -> tuple(item, item.getProductId())) .map(t -> tuple(t.r, lookup.byId((String) t.s))) .filter(t -> ((ProductInformation)t.s).isValid()) .collect( Collectors.toMap( t -> (OrderItem)t.r, t -> (ProductInformation)t.s ) ); if (map.keySet().size() != order.getItems().size()) { log.error("Some of the products in the order do not have product information, {} != {} ",map.keySet().siz map = null; } } return map; } We use a helper class, Tuple, which is nothing but two Object instances named r and s. We will list the code for this class later. It is very simple. In the streams expression, we first create the stream from the collection, and then we map the OrderItem elements to a stream of OrderItem and productId tuples. Then we map these tuples to tuples that now contain OrderItem and ProductInformation. These two mappings could be done in one mapping call, which would perform the two steps only in one. I decided to create the two to have simpler steps in each line in a vain hope that the resulting code will be easier to comprehend. The filter step is also nothing new. It just filters out invalid product information elements. There should actually be none. It happens if the order contains an order ID to a non-existent product. This is checked in the next statement when we look at the number of collected product information elements to see that all the items have proper information. The interesting code is how we collect the elements of the stream into a Map. To do so, we again use the collect method and also the Collectors class. This time, the toMap method creates the Collector. This needs two Function resulting expressions. The first one should convert the element of the stream to the key and the second should result in the value to be used in the Map. Because the actual type of the key and the value is calculated from the result of the passed lambda expressions, we explicitly have to cast the fields of the tuple to the needed types. Finally, the simple Tuple class is as follows: public class Tuple<R, S> { final public R r; final public S s; private Tuple(R r, S s) { this.r = r; this.s = s; } public static <R, S> Tuple tuple(R r, S s) { return new Tuple<>(r, s); } } There are still some classes in our code that deserve to be converted to functional style. These are the Checker and CheckerHelper classes. In the Checker class, we can rewrite the isConsistent method: public boolean isConsistent(Order order) { Map<OrderItem, ProductInformation> map = piCollector.collectProductInformation(order); if (map == null) { return false; } final Set<Class<? extends Annotation>> annotations = pcCollector.getProductAnnotations(order); return !checkers.stream().anyMatch( checker -> Arrays.stream( checker.getClass().getAnnotations() ).filter( annotation -> annotations.contains( annotation.annotationType()) ).anyMatch( x -> checker.isInconsistent(order) )); } Since you have already learnt most of the important stream methods, there is hardly any new issue here. We can mention the anyMatch method, which will return true if there is at least one element so that the Predicate parameter passed to anyMatch is true. It may also need some accommodation so that we could use a stream inside another stream. It very well may be an example when a stream expression is overcomplicated and needs to split up into smaller pieces using local variables. Finally, before we leave the functional style, we rewrite the containsOneOf method in the CheckHelper class. This contains no new elements and will help you check what you have learned about map, filter, flatMap, and Collector. Note that this method, as we discussed, returns true if order contains at least one of the order IDs given as strings: public boolean containsOneOf(String... ids) { return order.getItems().stream() .map(OrderItem::getProductId) .flatMap(itemId -> Arrays.stream(ids) .map(id -> tuple(itemId, id))) .filter(t -> Objects.equals(t.s, t.r)) .collect(Collectors.counting()) > 0; } We create the stream of the OrderItem objects, and then we map it to a stream of the IDs of the products contained in the stream. Then we create another stream for each of the IDs with the elements of the ID and one of the string IDs given as the argument. Then, we flatten these substreams into one stream. This stream will contain order.getItems().size() times ids.length elements: all possible pairs. We will filter out those pairs that contain the same ID twice, and finally, we will count the number of elements in the stream. Scripting in Java 9 We are almost ready with our sample program for this chapter. There is one issue, though it is not professional. When we have a new product that needs a new checker, we have to create a new release of the code. Programs in professional environments have releases. When the code is modified, bugs are fixed, or a new function is implemented, there are numerous steps that the organization requires before the application can go into production. These steps compose the release process. Some environments have lightweight release processes; others require rigorous and expensive checks. It is not because of the taste of the people in the organization, though. When the cost of a non-working production code is low and it does not matter if there is an outage or wrong functioning in the program, then the release process can be simple. This way, releases get out faster and cheaper. An example can be some chat program that is used for fun by the users. In such a situation, it may be more important to release new fancy features than ensuring bug-free working. On the other end of the palette, if you create code that controls an atomic power plant, the cost of failure can be pretty high. Serious testing and careful checking of all the features, even after the smallest change, can pay off. In our example, simple checkers may be an area that is not likely to induce serious bugs. It is not impossible but the code is so simple...Yes, I know that such an argument is a bit fishy, but let's assume that these small routines could be changed with less testing and in an easier way than the other parts of the code. How to separate the code for these little scripts, then, so that they do not require a technical release, a new version of the application, and not even restarting the application? We have a new product that needs a new check and we want to have some way to inject this check into the application environment without any service disruption. The solution we choose is scripting. Java programs can execute scripts written in JavaScript, Groovy, Jython (which is the JVM version of the language Python), and many other languages. Except JavaScript, the language interpreters of these languages are not a part of the JDK, but they all provide a standard interface, which is defined in the JDK. The consequence is that we can implement script execution in our code and the developers, who provide the scripts, are free to choose any of the available languages; we do not need to care to execute a JavaScript code. We will use the same API as to execute Groovy or Jython. The only thing we should know is what language the script is in. This is usually simple: we can guess that from the file extension, and if guessing is not enough, we can demand that the script developers put JavaScript into files with the .js extension, Jython into files with .jy or .py, Groovy into files with .groovy, and so on. It is also important to note that if we want our program to execute one of these languages, we should make sure that the interpreter is on the classpath. In the case of JavaScript, this is given; therefore, as a demonstration in this chapter, we will write our scripts in JavaScript. There will not be a lot; this is a Java book and not a JavaScript book after all. Scripting is usually a good choice when we want to pass the ability of programmatically configuring or extending our application. This is our case now. The first thing we have to do is to extend the production information. In case there is a script that checks the consistency of an order that a product is in, we need a field where we can specify the name of the script: private String checkScript; public String getCheckScript() { return checkScript; } public void setCheckScript(String checkScript) { this.checkScript = checkScript; } We do not want to specify more than one script per product; therefore, we do not need a list of script names. We have only one script specified by the name. To be honest, the data structure for the checker classes and the annotations, allowing multiple annotations per product and also per checker class, was too complicated. We could not avoid that, though, to have a complex enough structure that could demonstrate the power and capability of stream expressions. Now that we are over that subject, we can go on using simpler data structures focusing on script execution. We also have to modify the Checker class to not only use the checker classes but also the scripts. We cannot throw away the checker classes because, by the time we realize that we better need scripts for the purpose, we already have a lot of checker classes and we have no financing to rewrite them to be scripts. Well yes, we are in a book and not in real life, but in an enterprise, that would be the case. That is why you should be very careful while designing solutions for a corporate. The structures and the solutions will be there for a long time and it is not easy to throw a piece of code out just because it is technically not the best. If it works and is already there, the business will be extremely reluctant to spend money on code maintenance and refactoring. Summary: we modify the Checker class. We need a new class that can execute our scripts; thus, the constructor is modified: private final CheckerScriptExecutor executor; public Checker( @Autowired Collection<ConsistencyChecker> checkers, @Autowired ProductInformationCollector piCollector, @Autowired ProductsCheckerCollector pcCollector, @Autowired CheckerScriptExecutor executor ) { this.checkers = checkers; this.piCollector = piCollector; this.pcCollector = pcCollector; this.executor = executor; } We also have to use this executor in the isConsistent method: public boolean isConsistent(Order order) { final Map<OrderItem, ProductInformation> map = piCollector.collectProductInformation(order); if (map == null) { return false; } final Set<Class<? extends Annotation>> annotations = pcCollector.getProductAnnotations(order); Predicate<Annotation> annotationIsNeeded = annotation -> annotations.contains(annotation.annotationType()); Predicate<ConsistencyChecker> productIsConsistent = checker -> Arrays.stream(checker.getClass().getAnnotations()) .parallel().unordered() .filter(annotationIsNeeded) .anyMatch( x -> checker.isInconsistent(order)); final boolean checkersSayConsistent = !checkers.stream(). anyMatch(productIsConsistent); final boolean scriptsSayConsistent = !map.values(). parallelStream(). map(ProductInformation::getCheckScript). filter(Objects::nonNull). anyMatch(s -> executor.notConsistent(s,order)); return checkersSayConsistent && scriptsSayConsistent; } Note that in this code, we use parallel streams because, why not? Whenever it is possible, we can use parallel streams, even unordered, to tell the underlying system and also to the programmer fellows maintaining the code that order is not important. We also modify one of our product JSON files to reference a script instead of a checker class through some annotation: { "id" : "124", "title": "Desk Lamp", "checkScript" : "powered_device", "description": "this is a lamp that stands on my desk", "weight": "600", "size": [ "300", "20", "2" ] } Even JSON is simpler. Note that as we decided to use JavaScript, we do not need to specify the file name extension when we name the script. We may later consider further development when we will allow the product checker script maintainers to use different scripting languages. In such a case, we may still require that they specify the extension, and in case there is no extension, it will be added by our program as .js. In our current solution, we do not check that, but we may devote a few seconds to think about it to be sure that the solution can be further developed. It is important that we do not develop extra code for the sake of further development. Developers are not fortunetellers and cannot tell reliably what the future needs will be. That is the task of the business people. We put the script into the resource directory of our project under the scripts directory. The name of the file has to be powered_device.js because this is the name we specified in the JSON file: function isInconsistent(order){ isConsistent = false items = order.getItems() for( i in items ){ item = items[i] print( item ) if( item.getProductId() == "126" || item.getProductId() == "127" || item.getProductId() == "128" ){ isConsistent = true } } return ! isConsistent } This is an extremely simple JavaScript program. As a side note, when you iterate over a list or an array in JavaScript, the loop variable will iterate over the indexes of the collection or the array. Since I rarely program in JavaScript, I fell into this trap and it took me more than half an hour to debug the error I made. We have prepared everything we need to call the script. We still have to invoke it. To do so, we use the JDK scripting API. First, we need a ScriptEngineManager. This manager is used to get access to the JavaScript engine. Although the JavaScript interpreter has been a part of the JDK since Java 7, it is still managed in an abstract way. It is one of the many possible interpreters that a Java program can use to execute script. It just happens to be there in the JDK, so we do not need to add the interpreter JAR to the classpath. ScriptEngineManager discovers all the interpreters that are on the classpath and registers them. It does so using the Service Provider specification, which has been a part of the JDK for a long time, and by Java 9, it also got extra support in module handling. This requires the script interpreters to implement the ScriptEngineFactory interface and also to list the class that does it in the METAINF/services/javax.script.ScriptEngineFactory file. These files, from all the JAR files that are part of the classpath, are read as resources by ScriptEngineManager, and through this, it knows which classes implement script interpreters. The ScriptEngineFactory interface requires that the interpreters provide methods such as getNames, getExtensions, and getMimeTypes. The manager calls these methods to collect the information about the interpreters. When we ask a JavaScript interpreter, the manager will return the one created by the factory that said that one of its names is JavaScript. To get access to the interpreters through the name, file name extension or mime-type is only one of the functions of ScriptEngineManager. The other one is to manage Bindings. When we execute a script from within the Java code, we don't do it because we want to increase our dopamine levels. In the case of scripts, it does not happen. We want some results. We want to pass parameters and after the execution of the script, we want values back from the script that we can use in the Java code. This can happen in two ways. One is by passing parameters to a method or function implemented in the script and getting the return value from the script. This usually works, but it may even happen that some scripting language does not even have the notion of the function or method. In such a case, it is not a possibility. What is possible is to pass some environment to the script and read values from the environment after the script is executed. This environment is represented by Bindings. Bindings is a map that has String keys and Object values. In the case of most scripting languages, for example, in JavaScript, Bindings is connected to global variables in the script we execute. In other words, if we execute the following command in our Java program before invoking the script, then the JavaScript global variable, globalVariable, will reference the myObject object: myBindings.put("globalVariable",myObject) We can create Bindings and pass it to ScriptEngineManager but just as well we can use the one that it creates automatically, and we can call the put method on the engine object directly. There are two Bindings when we execute scripts. One is set on the ScriptEngineManager level. This is named global binding. There is also one managed by ScriptEngine itself. This is the local Bindings. From the script point of view, there is no difference. From the embedding side, there is some difference. In case we use the same ScriptEngineManager to create multiple ScriptEngine instances, then the global bindings are shared by them. If one gets a value, all of them see the same value; if one sets a value, all others will later see that changed value. The local binding is specific to the engine it is managed by. Since we only introduce Java scripting API in this book, we do not get into more details and we will not use Bindings. We are good with invoking a JavaScript function and to get the result from it. The class that implements the script invocation is CheckerScriptExecutor: package packt.java9.by.example.mybusiness.bulkorder.services; import ... @Component public class CheckerScriptExecutor { private static final Logger log = ... private final ScriptEngineManager manager = new ScriptEngineManager(); public boolean notConsistent(String script, Order order) { try { final Reader scriptReader = getScriptReader(script); final Object result = evalScript(script, order, scriptReader); assertResultIsBoolean(script, result); log.info("Script {} was executed and returned {}", script, result); return (boolean) result; } catch (Exception wasAlreadyHandled) { return true; } } The only public method, notConsistent, gets the name of the script to execute and also order. The latter has to be passed to the script. First it gets Reader, which can read the script text, evaluates it, and finally returns the result in case it is boolean or can at least be converted to boolean. If any of the methods invoked from here that we implemented in this class is erroneous, it will throw an exception, but only after appropriately logging it. In such cases, the safe way is to refuse an order. Actually, this is something that the business should decide. If there is a check script that cannot be executed, it is clearly an erroneous situation. In this case, accepting an order and later handling the problems manually has certain costs. Refusing an order or confirmation because of some internal bug is also not a happy path of the order process. We have to check which approach causes the least damage to the company. It is certainly not the duty of the programmer. In our situation, we are in an easy situation. We assume that the business representatives said that the order in such a situation should be refused. In real life, similar decisions are many times refused by the business representatives saying that it just should not happen and the IT department has to ensure that the program and the whole operation is totally bug free. There is a psychological reason for such a response, but this really leads us extremely far from Java programming. Engines can execute a script passed through Reader or as String. Because now we have the script code in a resource file, it seems to be a better idea to let the engine read the resource instead of reading it to a String: private Reader getScriptReader(String script) throws IOException { final Reader scriptReader; try { final InputStream scriptIS = new ClassPathResource( "scripts/" + script + ".js").getInputStream(); scriptReader = new InputStreamReader(scriptIS); } catch (IOException ioe) { log.error("The script {} is not readable", script); log.error("Script opening exception", ioe); throw ioe; } return scriptReader; } To read the script from a resource file, we use the Spring ClassPathResource class. The name of the script is prepended with the scripts directory and appended by the.js extension. The rest is fairly standard and nothing we have not seen in this book. The next method that evaluates the script is more interesting: private Object evalScript(String script, Order order, Reader scriptReader) throws ScriptException, NoSuchMethodException { final Object result; final ScriptEngine engine = manager.getEngineByName("JavaScript"); try { engine.eval(scriptReader); Invocable inv = (Invocable) engine; result = inv.invokeFunction("isInconsistent", order); } catch (ScriptException | NoSuchMethodException se) { log.error("The script {} thruw up", script); log.error("Script executing exception", se); throw se; } return result; } To execute the method in the script, first of all, we need a script engine that is capable of handling JavaScript. We get the engine from the manager by its name. If it is not JavaScript, we should check that the returned engine is not null. In the case of JavaScript, the interpreter is part of the JDK and checking that the JDK conforms to the standard would be paranoid. If ever we want to extend this class to handle not only JavaScript but also other types of scripts, this check has to be done, and also the script engine should probably be requested from the manager by the file name extension, which we do not have access to in this private method. But that is future development, not in this book. When we have the engine, we have to evaluate the script. This will define the function in the script so that we can invoke it afterwards. To invoke it, we need some Invocable object. In the case of JavaScript, the engine also implements an Invocable interface. Not all script engines implement this interface. Some scripts do not have functions or methods, and there is nothing to invoke in them. Again, this is something to do later, when we want to allow not only JavaScript scripting but also other types of scripting. To invoke the function, we pass its name to the invokeFunction method and also the arguments that we want to pass on. In this case, this is the order. In the case of JavaScript, the integration between the two languages is fairly developed. As in our example, we can access the field and the methods of the Java objects that are passed as arguments and the returned JavaScript true or false value is also converted to Boolean magically. There are some situations when the access is not that simple though: private void assertResultIsBoolean(String script, Object result) { if (!(result instanceof Boolean)) { log.error("The script {} returned non boolean", script); if (result == null) { log.error("returned value is null"); } else { log.error("returned type is {}", result.getClass()); } throw new IllegalArgumentException(); } } } The last method of the class checks that the returned value, which can be anything since this is a script engine, is convertible to boolean. It is important to note that the fact that some of the functionality is implemented in script does not guarantee that the application works seamlessly. There may be several issues and scripts may affect the inner working of the entire application. Some scripting engines provide special ways to protect the application from bad scripts, others do not. The fact that we do not pass but order to the script does not guarantee that a script cannot access other objects. Using reflection, static methods, and other techniques there can be ways to access just anything inside our Java program. We may be a bit easier with the testing cycle when only a script changes in our code base, but it does not mean that we should blindly trust any script. In our example, it probably would be a very bad idea to let the producers of the products upload scripts to our system. They may provide their check scripts, but these scripts have to be reviewed from the security point of view before being deployed into the system. If this is properly done, then scripting is an extremely powerful extension to the Java ecosystem, giving great flexibility to our programs. Summary In this chapter, we have developed the ordering system of our enterprise application. Along with the development of the code, we met many new things. You learned about annotations and how they can be handled by reflections. Although not strongly related, you learned how to use lambda expressions and streams to express some programming constructs simpler than conventional loops. In the last part of the chapter, we extended the application using scripting, by invoking JavaScript functions from Java and also by invoking Java methods from JavaScript. In fact, with all this knowledge, we matured to a Java level that is needed for enterprise programming. The rest of the topics the book covers are for the aces. But you want to be one, don't you? This is why I wrote the rest of the chapters. Read on! Building an Accounting Application Using Reactive Programming In this chapter, we will develop a sample program that does the inventory management part of the company we created the order handling code for. Do not expect a fully developed, ready-to-use, professional application, and also, do not expect that we will get into the details of accounting and bookkeeping. That is not our aim. We will focus more on the programming technique that is of our interest —reactive programming. Sorry pals, I know that bookkeeping and accounting is fun, but this is not that book. Reactive programming is an old (well, what is old in computer science?) approach that has come recently to Java. Java 9 is the first release that supports some of the aspects of reactive programming in the standard JDK. In one sentence, reactive programming is about focusing more on how the data flows and less on how the implementation handles the data flow. As you may recall, this is also a step towards describing what we want to do from the description of how to do it. After going through this chapter, you will understand what reactive programming is and what tools there are in Java that you can utilize. You will also understand what reactive programming is good for and when and how you can utilize this principle in the future, as there will be more and more frameworks supporting reactive programming in Java. In this chapter, you will learn the following topics: Reactive programming in general Reactive streams in Java How to implement our sample code in a reactive way Reactive... what? There are reactive programming, reactive systems, and reactive streams. These are three different things related to each other. It is not without reason that all the three are called reactive. Reactive programming is a programming paradigm similar to object-oriented programming and functional programming. A reactive system is a system design that sets certain aims and technological constraints on how a certain type of information systems should be designed to be reactive. There are a lot of resemblances to reactive programming principles in this. A reactive stream is a set of interface definitions that helps to achieve similar coding advantage to reactive systems and which can be used to create reactive systems. Reactive stream interfaces are a part of JDK 9, but are available not only in Java, but also in other languages. We will look at these in separate sections, at the end of which, you will presumably have a better understanding of why each of them is called reactive. Reactive programming in a nutshell Reactive programming is a paradigm that focuses more on where the data flows during computation than on how to compute the result. If the problem is best described as several computations that depend on the output of each other but several may be executed independent of the other, reactive programming may come into the picture. As a simple example, we can have the following computation that calculates the value of h from some given b, c, e, and f values, using f1, f2, f3, f4, and f5 as simple computational steps: a d k g h = = = = = f1(b,c) f2(e,f) f3(e,c) f4(b,f,k) f5(d,a,g) If we write these in Java in the conventional way, the methods f1 to f5 will be invoked one after the other. If we have multiple processors and are able to parallelize the execution, we may perform some of the methods in parallel. This, of course, assumes that these methods are purely computational methods and do not change the state of the environment, and, in this way, can be executed independent of each other. For example, f1, f2, and f3 can be executed independent of each other. The execution of the function f4 depends on the output of f3, and the execution of f5 depends on the output of f1, f2, and f4. If we have two processors, we can execute f1 and f2 together, followed by the execution of f3, then f4, and finally, f5. These are four steps. If we look at the preceding calculation not as commands but rather as expressions and how the calculations depend on each other, then we do not dictate the actual execution order and the environment may decide to calculate f1 and f3 together, then f2 and f4, and finally f5, saving one step. This way, we can concentrate on the data flow and let the reactive environment act upon it without putting extra constraints. This is a very simple approach of reactive programming. The description of the calculation in the form of expressions gives the data flow, but in the explanation, we still assumed that the calculation is executed synchronously. If the calculations are executed on different processors that are on different machines connected to a network, then the calculation may not and does not need to be synchronous. Reactive programs can be asynchronously executed if the environment is asynchronous. It may happen that the different calculations, f1 to f4, are implemented and deployed on different machines. In such a case, the values calculated are sent from one to the other over the network and the nodes execute the calculation every time there is a change in the inputs. This is very similar to good old analog computers that were created using simple building blocks and the calculations were done using analogue signals. The program was implemented as an electronic circuit, and when the input voltage or current (usually voltage) changed in the inputs, the analog circuits followed it in light's speed, and the result appeared in the output. In such a case, the signal propagation was limited by the speed of light on the wires and analog circuitry speed in the wired modules, which was extremely fast and may beat digital computers. When we talk about digital computers the propagation of the signal is digital, and this way, it needs to be sent from one calculation node to the other one, be it some object in JVM or some program on the network. A node has to execute its calculation if: Some of the values in the input have changed The output of the calculation is needed If the input has not changed, then the result should eventually be the same as the last time; thus, the calculation does not need to be executed again—it would be a waste of resources. If the result of the calculation is not needed, then there is no need to perform the calculation even if the result would not be the same as the last one. No one cares. To accommodate this, reactive environments implement two approaches to propagate the values. The nodes may pull the values from the output of other modules. This will ensure that no calculation that is not needed will be executed. The modules may push their output to the next module that depends on them. This approach will ensure that only changed values ignite calculation. Some of the environments may implement a hybrid solution. When values change in the system, the change is propagated towards the other nodes that again propagate the changes to another node and so on. If we imagine the calculation dependencies as a directed graph, then the changes travel towards the transitive closure of the changed values along the nodes connected. The data may travel with all the values from one node output to the other node input or only the change may travel. The second approach is more complex because it needs the changed data and also meta information that describes what has changed. On the other hand, the gain may be significant when the output and input set of data is huge and only a small portion of it is changed. It may also be important to calculate and propagate only the actual delta of the change when there is a high probability that some of the nodes do not change the output for many of the different inputs. In such a case, the change propagation may stop at the node where there is no real change in spite of the changed input values. This can save up a lot of calculation in some of the networks. In the configuration of the data propagation, the directed acyclic graph can be expressed in the code of the program, it can be configured or it can even be set up and changed during the execution of the code dynamically. When the program code contains the structure of the graph, the routes and the dependencies are fairly static. To change the data propagation, the code of the program has to be changed, recompiled, and deployed. In the case of multiple network node programs, this may even need multiple deployments that should be carefully furnished to avoid different incompatible versions running on different nodes. There should be similar considerations when the graph is described in some configuration. In such a case, the compilation of the program(s) may not be needed when only the wiring of the graph is changed, but the burden to have compatible configuration on different nodes in the case of a network execution is still there. Letting the graph change dynamically also does not solve this problem. The setup and the structure are more flexible and, at the same time, more complex. The data propagated along the edges of the graph may contain not only computational data but also data that drives change in the graph. Many times, this leads to a very flexible model called higher-order reactive programming. Reactive programming has a lot of benefits but, at the same time, may be very complex, sometimes too complex, for simple problems. It is to be considered when the problem to be solved can easily be described using data graph and simple data propagations. We can separate the description of the problem and the order of the execution of the different blocks. This is the same consideration that we discussed in the previous chapter. We describe more about the what to do part and less about the how to do part. On the other hand, when the reactive system decides the order of execution, what is changed, and how that should be reflected on the output of other blocks, it should do so without knowing the core of the problem that it is solving. In some situations, coding the execution order manually based on the original problem could perform better. This is similar to the memory management issue. In modern runtime environments, such as the JVM, Python runtime, Swift programming, or even Golang, there is some automated memory management. When programming in C, the programmer has full control over memory allocation and memory release. In the case of real-time applications, where the performance and response time is of the utmost importance, there is no way to let an automated garbage collector take time and delay the execution from time to time. In such a case, the C code can be optimized to allocate memory when needed; there is a resource for the allocation and release of memory when possible and there is time to manage memory. These programs are better performing than the ones created for the same purpose using a garbage collector. Still, we do not use C in most of the applications because we can afford the extra resource needed for automated memory collection. Even though it would be possible to write a faster code managing the memory manually, automated code is faster than what an average programmer would have created using C, and also, the frequency of programming errors is much lower. Just as there are some issues that we have to pay attention to when using automated memory management, we have to pay attention to some issues in a reactive environment, which would not exist in the case of manual coding. Still, we use the reactive approach for its benefits. The most important issue is to avoid loops in the dependency graph. Although it is absolutely perfect to write the definition of calculations, a reactive system would probably not be able to cope with these definitions. Some reactive systems may resolve in some simple-case cyclic redundancy, but that is some extra feature and we generally just have to avoid that. Consider the following computations: a = b + 3 b = 4 / a Here, a depends on b, so when b changes, a is calculated. However, b also depends on a, which is recalculated and, in this way, the system gets into an infinite loop. The preceding example seems to be simple, but that is the feature of a good example. Real-life problems are not simple, and in a distributed environment, it is extremely hard sometimes to find cyclic redundancy. Another problem is called glitch. Consider the following definition: a = b + 3 q = b + a When the parameter b is changed, for example, from 3 to 6, the value of a will change from 6 to 9, and thus, q will change from 9 to 15. This is very simple. However, the execution order based on the recognition of the changes may first alter the value of q from 9 to 12 before modifying it to 15 in a second step. This can happen if the calculating node responsible for the calculation of q recognizes the change in b before the value of a as a consequence of the change in the value of b. For a short period of time, the value of q will be 12, which does not match the previous and also does not the changed state. This value is only a glitch in the system that happens after an input changes and also disappears without any further change in the input in the system. If you have ever learnt the design of logical circuits, then static hazards may ring a bell. They are exactly the same phenomenon. Reactive programming also assumes that the calculations are stateless. The individual nodes that perform the calculation may have a state in practice and, many times, they do. It is not inherently evil to have a state in some calculation. However, debugging something that has a state is significantly more complex than debugging something that is stateless, functional. It is also an important aid to the reactive environment, letting it perform different optimizations based on the fact that the calculations are functional. If the nodes have a state, then the calculations may not be rearranged freely because the outcome may depend on the actual evaluation order. These systems may not really be reactive, or, at least, it may be debated. Reactive systems Reactive system is defined in the reactive manifesto at http://www.reactivemanifesto.org/. The creators of the manifesto realized that with the change of technology, new system patterns will need to be developed in enterprise computing to leverage the new technology and yield better outcomes. The manifesto envisions systems that are: Responsive Resilient Elastic Message-driven. The first three features are user values; the last one is more of a technological approach to get the values. Responsive A system is responsive if it gives results in a reliable manner. If you talk to me, I will answer your question or, at least, tell you that I do not know the answer or that I was not able to understand the question. Better if you get the answer, but if a system cannot give that to you it is still expected to give something back. If you have past experience with client operating systems from just ten years ago and some old computers, you can understand this. Getting a rotating hourglass is frustrating. You just do not know whether the system is working to get you the answer or is totally frozen. A reactive system has to be responsive. The response should come in a timely manner. The actual timing depends on the actual system. It may be milliseconds, seconds, or even hours in case the system is running on a space ship travelling towards the other side of Jupiter. The important thing is that the system should guarantee some soft upper limit for the response time. This does not necessarily mean that the system should be a real-time solution, which is a much stricter requirement. The advantage of responsiveness is not only that the user does not become nervous in front of the computer. After all, most of these services are used by other services that mainly communicate with each other. The real advantage is that error discovery is more reliable. If a reactive system element becomes non responsive, it is certainly an error condition, and something should be done about it, out of the scope of normal operations (replace a faulty communication card, restart a system, and so on). The sooner we can identify an error state, the cheaper it is to fix it. The more we can identify where the problem is, the less time and money we could spend localizing the error. Responsiveness is not about speed. It is about better operation, better quality. Resilient Resilient systems keep working even when there is some error. Well, not any error. That would be a miracle, or simply nonsense! An error generally is an error. If the Armageddon comes and it is the end of the world as we know it, even resilient systems will not be responsive. For smaller disruptions, however, there may be some cure to make the systems resilient. There are techniques that may help if only a disk fails, there is a power outage, or there is a programming error. Systems may be replicated, so when one of the instances stops responding, some other instance may take up the task of the failing one and can go on working. Systems prone to errors may be isolated from each other in terms of space or time. When there is an earthquake or flood at one location, the other location may still go on working. If different components do not need to communicate in real time and messages are stored and forwarded in a reliable manner, then this is not a problem even if the two systems are never available at the same time. They can still cooperate taking up the messages, performing the task they are supposed to, and sending out the resulting message afterwards. Errors in the system have to be addressed even if the system remains responsive. Errors do not affect the responsiveness of a resilient system, but the level of resilience decreases and should be restored. Elastic Elasticity means that the system is adapting to the load. We can have a huge system, with lots of processors capable of serving the largest anticipated demand. That is not elasticity. Since the demand is not constant and, most of the time, is smaller than the maximum, the resources of such a system are idle. This causes waste of time, CPU cycle, energy, and, thus, ecological footprint. Having systems run on the cloud can avoid such losses. The cloud is nothing but many computers that somebody operates for multiple applications, for multiple corporations even, and each rents only the CPU cycles that it really needs and only when it needs. Other times, when the load is smaller the CPU and the electric power can be used by someone else. Since different applications and different corporations have different peak times, the loss of resources is less with this model. There are many issues that have to be solved, such as data isolation and protection of information from eavesdropping, but these are mainly solved. Secret service corporations will not rent resources from a cloud service to run their computations (perhaps, they'd do for some other purpose) and some other paranoid companies may also refrain from doing that, but most of the companies will do. It is more effective and is thus cheaper even after considering all the side effects one can consider. Elasticity means that the allocated resources follow, or rather anticipate the coming needs. When the system anticipates higher capacity needs, it allocates more resources and at off-peak time, it releases the resources so that other cloud customers can use it. Elasticity also assumes that the system is scalable. The two things, elasticity and scalability, are closely related but are not the same. Scalability means that the application can accommodate higher load, allocating more resources. Scalability does not care whether this allocation is static buying and powering of huge computer boxes in a computing center dedicated to the application or dynamic allocation of resources from the cloud on demand. Scalability simply means that if the demand doubles, then the resources can also be multiplied to meet the demand. If the multiplication factor in the resources needed is the same or is not more than the factor in demand, then the application is scalable. If we need more resources to meet the demand, or if we cannot meet the demand even if the demand increases only moderately, then the application is not scalable. Elastic applications are always scalable; otherwise, they cannot be elastic. Message-driven Reactive systems are message-driven; not because we need message-driven systems but much rather because message-driven systems are those that can deliver responsiveness, resilience, and elasticity at the same time. Message-driven architecture means that the information travels between the components disconnected. One component sends a message and then forgets it. It does not wait for the other component to act upon the message. When the message is sent, all the tasks on behalf of the sending component are performed and all the resources needed to handle the tasks are released, resulting in the message being released and ready to be used for the next task. Message-driven does not necessarily mean networking. Messages can travel between objects, threads, and processes inside the same machine. On the other hand, if the interfaces to the messaging architecture are well-designed, then the components do not need to be modified if the infrastructure changes and the messages that were previously passing between threads will now have to travel through the ocean in IP packets. Sending messages makes it possible to isolate the sender and the receiver in space and time, just as we described, as a means for elasticity. The receiver may pick up the message some time after it arrived, when it has the resources to do so. Responsiveness, though, requires that this time is not in the unreachable distant future but in some limited distance. If the message cannot be processed successfully, another message may signal the error. An error message is not the result we expect, but it is still some response and the system remains responsive with all the benefits it means. Back-pressure Message handling, with the appropriate messaging interfaces and implementation, supports back-pressure. Back-pressure is a means to lessen the burden on a component when it cannot or can barely handle more messages. Messages may be queuing for processing, but no real-life queue has unlimited capacity and reactive systems should not lose a message uncontrolled. Back-pressure signals the load of the component to the message producers, asking them to lessen the production. It is like a water pipe. If you start closing the outlet of the pipe, the pressure starts to increase in the pipe backward, the water source forcing it to deliver less and less water. Back-pressure is an effective way of handling load because it moves load handling to the component that can really do it. In old-fashioned queuing systems, there is a queue that stores the items till the component receiving them can consume them, doing its job. A queue design can be good if there is a well-defined limit for the size of the load and for the maximum size of the queue. If ever the queue is full, the items cannot be delivered and the system stalls. Applying back-pressure is a bit different. A queue may still be used in front of the components for performance optimization and to ensure responsiveness. The producer of the item can still put the produced item in the queue and return to attending to its own duties and does not need to wait till the consumer can attend to the item. This is decoupling, as we mentioned earlier. Seeing that the queue is full or almost full can also act as a very simple back-pressure. It is not true if someone says that queues are totally missing this feature. At times, it may simply be totally sufficient just to look at the capacity of a queue, and also the items in it, to see if there is some need to lessen the load on the receiver the queue belongs to. But the producer does this, not the receiver, and that is an essential problem. The producer sees that the receiver is not keeping pace with the supply but the producer does not have any information about the cause, and not knowing the cause cannot predict the future behavior. Having a backpressure information channel from the receiver to the producer makes the story more fine grained. The producer may see that there are, say, 10 slots in the queue and it thinks that there is no problem; the producer decides to deliver eight more items in the next 150ms. One item usually takes 10ms to process, give or take; thus the items are expected to be processed in less than 100ms, which is just better than the required 200ms maximum. The producer only knows that an item usually takes 10ms to process. The receiver, on the other hand, sees that the last item it got into the queue requires so much processing that, by itself, it will require 200ms. To signal this, it can tell the producer over the back-pressure not to deliver new items till further notice. The receiver knows that the items would have fit in the queue fine but would not be processed in a timely manner. Using this information, the producer will issue some commands to the cloud control to allocate another processing and sends the next eight items to the new receiver, letting the old one do its cumbersome job it has to with that far above than average item. Back-pressure lets you aid the data load control, with information created by the receivers that have the most information about processing the items. Reactive streams Reactive streams started as an initiative to provide a standard of handling data streams in an asynchronous mode by regulating the push of the data using back-pressure. The original site of the project is http://www.reactive-streams.org/. Reactive streams are now implemented in JDK 9 in the java.util.concurrent package. The aim of the definition of reactive streams is to define the interface that can handle the propagation of the generated data in a totally asynchronous way without the need on the receiving side to buffer the unlimited created data. When data is created in a stream and is made available to be worked on the worker that gets the data, has to be fast enough to handle all the data that is generated. The capacity should be high enough to handle the highest production. Some intermediate buffers may handle peaks, but if there is no control that stops or delays production when the consumer is at the top of its capacity, the system will fail. Reactive system interfaces are designed to provide a way to support back-pressure. Backpressure is a process to signal the producer of the data to slow down or even to stop the production to the level that fits the consumer. Every call the interfaces define is asynchronous so that the performance of one part is not affected by the delays in the execution of other parts. The initiative did not aim to define the way in which data is transferred between production and consumption. It focuses on the interfaces to give a clear structure for the programs and also to give an API that will work with all the implementations. Reactive programming in Java Java is not a reactive language. This does not mean that we cannot create reactive programs in Java. There are libraries that support different reactive programming approaches. It is to mention the Akka framework and the ReactiveX that also exist for other languages as well. With Java 9, the JDK starts to support reactive programming, providing a few classes and interfaces for the purpose. We will focus on these features. The JDK contains the java.util.concurrent.Flow class, which contains related interfaces and some static methods to support flow controlled programs. The model that this class supports is based on Publisher, Subscriber, and Subscription. As a very simple explanation, a Publisher accepts a subscription from a Subscriber. A Subscriber gets the data it subscribed to when the data is available. The interfaces focus on the very core of the data flow control of the communication and are a bit abstract. No surprise, they are interfaces. However, it may not be simple to understand their working at first. The Publisher interface defines the subscribe method. This is the only method this interface defines and that is because this is the only thing that a real publisher can be asked. You can subscribe to the publications. The argument of the method is a Subscriber that subscribes to the publications: void subscribe(Flow.Subscriber<? super T> subscriber) There is a readily available Publisher class in the JDK that we will look at later. When the subscribe method of the Publisher is called, it has to decide if the subscriber can get the subscription or not. Usually, the subscription is accepted but the implementation has the freedom to refuse a subscription attempt. Publisher may refuse a subscription if, for example, the subscription for the actual subscriber was already performed and the Publisher implementation does not allow multiple subscriptions from the same subscriber. The implementation of the method is required to call the onError method of subscriber, with Throwable as the argument. In the case of multiple subscriptions, IllegalStateException seems to be suitable, as the JDK documentation defines at the moment. If the subscription is successful, Publisher is expected to call the onSubscribe method of subscriber. The argument to this method is a Subscription object (an instance of a class that implements the interface Subscription). This way, the Publisher notifies the Subscriber that the subscription request was accepted, and also passes an object to manage the subscription. Managing the subscription as an abstraction could be imagined as a complex task, but in the case of reactive streams, it is very simple. All the subscriber can and should do is to set the number of items it can receive at the moment, and the subscription can be cancelled. Why should the Publisher call back the onSubscribe method of Subscriber? Why doesn't it simply return the subscription or throw some error? The reason for this complex behavior is that it may not be the Subscriber that invokes the subscribe method. Just as in real life, I can subscribe and pay for a year of a magazine subscription as a Christmas gift. (This is the season when I am writing this part of the book.) In our code, some wiring component responsible for who is notified about a certain data change calls subscribe and not necessarily the subscriber. The Subscriber is only responsible for the minimal things that a subscriber should be responsible for. The other reason is that the whole approach is asynchronous. When we subscribe to something, the subscription may not be available and ready immediately. There could be some long-running processes that need to finish till the subscription will be available and the caller that is calling subscribe does not need to wait for the completion of the process. When the subscription is ready it is passed to the subscriber, to the very entity that really needs it. The Subscriber interface defines the onSubscribe, onError (we have already talked about these), onComplete and onNext methods. It is important in the definition of these interfaces that the subscriber gets the items from Publisher or from some other object to which the Publisher delegates this task via some push. The subscriber does not need to go to the newsstand to get the next issue; someone calling the onNext method delivers the issue to it directly. This also bears the consequence that unless there are some controls in the hands of the Subscriber, it could happen that the Publisher floods the Subscriber with items. Not every Subscriber is capable of handling unlimited items. The Subscriber gets a Subscription object upon performing the subscription and this object can be used to control the flow of the item objects. The Publisher creates the Subscription object and the interface defines two methods: cancel and request. The cancel method should be called by the Subscriber to notify the Publisher that it should not deliver more items. The subscription is cancelled. The request(long n) method specifies that the subscriber is prepared to get at most n items via subsequent calls to the onNext method: If the subscriber has already invoked the request method, the specified number is added to the subscription counter. In other words, the specified long value does not reflect the actual state of the subscriber. It is a delta, increasing some counters maintained by the Publisher that counts the number of items that can be delivered adding the value of the long argument and decrementing by one on each item delivered to the Subscriber. The most usual approach is to call request(1) each time the Subscriber has finished processing a request. If the request method is invoked with the Long.MAX_VALUE argument, the Publisher may just send any item that it can without counting and without limit. This is essentially switching off the back-pressure mechanism. The specification also mentions that the call to cancel does not necessarily mean that there will be no more issues delivered at all. Cancellation is done on best effort. Just as in real life, when you send your mail to the daily paper with your intent to cancel the subscription, the publisher will not send an agent to stop the postman before he drops the issue to your mailbox. If something was already on its way when the cancellation arrived to the publisher it goes its way. If the Publisher has already started some asynchronous process that cannot reasonably be stopped, then the method onNext method will be invoked with some of the elements. The Publisher and Subscriber interfaces have a generic parameter, T. This is the type of items that the Publisher interface publishes and the Subscriber interface gets in the onNext method. To be a bit more precise, the Subscriber interface can have an R type, which is a superclass of T; thus, it is compatible with the Publisher interface. For example, if Publisher publishes Long values, then the Subscriber interface can accept Long, Number, or Object in the argument of the onNext method, depending on the declaration of the class that implements Subscriber. The Flow class also contains a Processor interface that extends both Publisher and Subscriber. This interface is there to be implemented by classes that also accept data and send data to other components in the reactive flow. Such elements are very common in reactive stream programs because many elements that perform some tasks get the items to work on from other reactive stream elements; thus, they are Subscribers and, at the same time, they send it after they have finished their tasks; thus, they are Publishers. Implementing inventory Now that we have discussed a lot of technologies and programming approach, it is very much the time to implement some sample code. We will implement inventory keeping in our application using reactive streams. For the example, the inventory will be very simple. It is a Map<Product,InventoryItem> that holds the number of items for each product. The actual map is ConcurrentHashMap and the InventoryItem class is a bit more complex than a Long number to properly handle concurrency issues. When we design a program that is built on responsive streams, we do not need to deal with much concurrency locking, but we still should be aware that the code runs in a multithread environment and may exhibit strange behavior if we do not follow some rules. The code for the Inventory class is fairly simple since it handles only a map: package packt.java9.by.example.mybusiness.inventory; import ...; @Component public class Inventory { private final Map<Product, InventoryItem> inventory = new ConcurrentHashMap<>(); private InventoryItem getItem(Product product) { inventory.putIfAbsent(product, new InventoryItem()); return inventory.get(product); } public void store(Product product, long amount) { getItem(product).store(amount); } public void remove(Product product, long amount) throws ProductIsOutOfStock { if (getItem(product).remove(amount) != amount) throw new ProductIsOutOfStock(product); } } The inventory item maintaining the class is a bit more complex since this is the level where we handle a bit of concurrency or, at least, this is the class where we have to pay some attention: package packt.java9.by.example.mybusiness.inventory; import java.util.concurrent.atomic.AtomicLong; public class InventoryItem { private final AtomicLong amountOnStock = new AtomicLong(0); void store(long n) { amountOnStock.accumulateAndGet(n, (stock, delta) -> stock + delta); } long remove(long delta) { class ClosureData { long actNr; } ClosureData d = new ClosureData(); amountOnStock.accumulateAndGet(delta, (stock, n) -> stock >= n ? stock - (d.actNr = n) : stock - (d.actNr = 0) ); return d.actNr; } } When we add products to the inventory, we have no limit. The storage shelves are extremely huge and we do not model that they once may get full and the inventory may not be able to accommodate more items. When we want to remove items from the repository, however, we have to deal with the fact that there may not be enough items from the product. In such a case, we do not remove any items from the repository. We serve the customer to full satisfaction or we do not serve at all. To maintain the number of the items in the inventory, we use AtomicLong. This class has the accumulateAndGet method. This method gets a Long parameter and a LongBinaryOperator that we provide in our code as a lambda. This code is invoked by the accumulateAndGet method to calculate the new value of the stock. If there are enough items, then we remove the requested number of items. If there are not enough items on stock, then we remove zero. The method returns the number of items that we actually return. Since that number is calculated inside the lambda, it has to escape from there. To do so, we use ClosureData defined inside the method. Note that, for example, in Groovy we could simply use a Long d variable and alter the variable inside the closure. Groovy calls lambda to closures, so to say. In Java we cannot do so because the variables that we can access from inside the method should be effectively final. However, this is nothing more than a bit more explicit notation that belongs to the closure environment. The ClosureData d object is final as opposed to the field the class has, which can be modified inside the lambda. The most interesting class that we are really interested in this chapter is InventoryKeeper. This class implements the Subscriber interface and is capable of consuming orders to maintain the inventory: package packt.java9.by.example.mybusiness.inventory; import ... public class InventoryKeeper implements Flow.Subscriber<Order> { private static final Logger log = LoggerFactory.getLogger(InventoryKeeper.class); private final Inventory inventory; public InventoryKeeper(@Autowired Inventory inventory) { this.inventory = inventory; } private Flow.Subscription subscription = null; private static final long WORKERS = 3; @Override public void onSubscribe(Flow.Subscription subscription) { log.info("onSubscribe was called"); subscription.request(WORKERS); this.subscription = subscription; } The onSubscribe method is invoked after the object is subscribed. The subscription is passed to the object and is also stored in a field. Since the subscriber needs this subscription in subsequent calls, when an item passed in onNext is processed and a new item is acceptable, a field is a good place to store this object in. In this method, we also set the initial request to three items. The actual value is simply demonstrative. Enterprise environments should be able to configure such parameters: private ExecutorService service = Executors.newFixedThreadPool((int) WORKERS); The most important part of the code is the onNext method. What it does is actually goes through all the items of the order and removes the number of items from the inventory. If some of the items are out of stock, then it logs an error. This is the boring part. The interesting part is that it does this through an executor service. This is because the call to onNext should be asynchronous. The publisher calls onNext to deliver the item, but we should not make it wait for the actual processing. When the postman brings your favorite magazine, you don't start reading it immediately and make the postman wait for your signature approving acceptance. All you have to do in onNext is fetch the next order and make sure that this will be processed in due time: @Override public void onNext(Order order) { service.submit(() -> { int c = counter.incrementAndGet(); for (OrderItem item : order.getItems()) { try { inventory.remove(item.getProduct(), item.getAmount()); } catch (ProductIsOutOfStock exception) { log.error("Product out of stock"); } } subscription.request(1); counter.decrementAndGet(); } ); } @Override public void onError(Throwable throwable) { log.info("onError was called for {}", throwable); } @Override public void onComplete() { log.info("onComplete was called"); } } The actual implementation in this code uses ThreadPool with three threads in it. Also, the number of required items is three. This is a logical coincidence: each thread works on a single item. It does not need to be like that, even if in most cases it is. Nothing can stop us from making more threads working on the same item if that makes sense. The opposite is also true. One single thread may be created to work on multiple items. These codes will probably be more complex and the whole idea of these complex execution models is to make the coding and the logic simpler, move the multithreading, coding, and implementation issues into the framework, and focus on the business logic in the application code. But I cannot tell that there may not be an example for a subscriber working multiple threads on multiple items, intermingled. The last code we have to look at in this chapter is the unit test that drives the code with some examples: public void testInventoryRemoval() { Inventory inventory = new Inventory(); SubmissionPublisher<Order> p = new SubmissionPublisher<>(); We create Publisher using the JDK class, SubmissionPublisher, which neatly implements this interface delivering multithread functionality for us without much hassle: p.subscribe(new InventoryKeeper(inventory)); We create an inventory keeper and we subscribe to the publisher. This does not start delivering anything because there are no publications yet, but it creates a bond between the subscriber and the publisher telling, them that whenever there is a product submitted, the subscriber wants it. After that, we create the products and store them in the inventory, 20 pieces altogether, and we also create an order that wants 10 products to be delivered. We will execute this order many times. This is a bit of simplification, but for the test, there is no reason to create separate order objects that have the same products and the same amounts in the list of items: Product product = new Product(); inventory.store(product, 20); OrderItem item = new OrderItem(); item.setProduct(product); item.setAmount(10); Order order = new Order(); List<OrderItem> items = new LinkedList<>(); items.add(item); order.setItems(items); After all this has been done, we submit the order to the Publisher 10 times. It means that there are 10 orders for the same product, each asking for 10 pieces, that is, 100 pieces together. Those are 100 pieces against the warehouse where we have only 20 of it. What we should expect is that only the first two orders will be fulfilled and the rest will be rejected and that is what will actually happen when we execute this code: for (int i = 0; i < 10; i++) p.submit(order); log.info("All orders were submitted"); After all the orders are published, we wait for half a second so that the other threads have time to execute and then we finish: for (int j = 0; j < 10; j++) { log.info("Sleeping a bit..."); Thread.sleep(50); } p.close(); log.info("Publisher was closed"); } Note that this is not a regular unit test file. It is some test code to play around, which I also recommend for you to execute, debug, and look at the different log outputs. Summary In this short chapter, we had a look at reactive programming, reactive systems, and reactive streams. We discussed the similarities and the differences between these that may lead to confusions. We paid special attention to Java 9 reactive streams that have practically nothing to do with Stream classes and methods. In the second half of the chapter, we discussed a very simple example that uses reactive streams. After reading this chapter, you have learned a lot about the Java language and programming. We did not detail all the small bits of Java, but that is not possible in a book. I dare say that there is no man (or woman for that matter) on the Earth or around it on an orbital route, wherever humans are, who knows everything about Java. We, however, know enough by now to start coding in an enterprise environment and to learn more and more on the go till we retire, or even after that. What is still left is a little bit of programming. In the previous sentence I said coding to make some distinction. Coding is not the same as programming. Coding is a technique used in the profession of programming. During the next, and last, chapter we will see the aspects of programming and how it can, and should, be done in professional manner. This is rarely a part of an introductory book, but I am happy that we could agree on this topic with the publisher. This way, you can finish the book not only with the knowledge that you learn from this book, but also with a vision, looking ahead on the road you will walk up the hillside to the top. You will know the topics, areas, and subjects that you can go on learning. Finalizing Java Knowledge to a Professional Level By now, you have learned the most important areas and topics needed for a professional Java developer. What we still have ahead of us in this book is to discuss some topics that will lead you from being a junior developer to a senior developer. Reading this chapter will not make anyone a senior developer, though. The previous chapters were the roads that we walked through. This chapter is only the map. If each of the previous chapters covered a short walk of a few miles in the journey of coding to reach the harbor, then this chapter is the nautical map to discover a new continent. We will briefly bite into some very deep and high-level professional areas, such as creating a Java agent, compile-time annotation processing, polyglot programming, a bit of architecture design and tools, and techniques to work in teams. We'll do it just for the taste. Now, you have enough knowledge to understand the importance of these topics, and getting a taste will create an appetite for the coming years of selfdevelopment, or, at least, that is my intention to make you, the reader, addicted. Java deep technologies In this section, we will list three technologies: Java agent Polyglot programming Annotation processing Knowing them is not a must for a Java professional. Knowing about them is. Java agents are used mainly in development environments and in operation. They are complex runtime technologies that interact with the already running JVM. Annotation processing is the other end. Annotation processors are plugged into the Java compiler. Polyglot programming is in the middle. It is JVM programming, just like programming in Java, but by using some different language or, perhaps, some different language and Java together. Or even many languages, such as Jython, Groovy, Clojure, and Java together. We will discuss these technologies so that we will get some idea about what they are and where to look for further information in case we want to learn more about them. Java agent A Java agent is a Java program that is loaded by the Java runtime in a special way and can be used to interfere with the byte code of the loaded classes, altering them. They can be used to: List or log, and report the loaded classes during runtime, as they are loaded Modify the classes so that the methods will contain extra code to report runtime behavior Support debuggers to alter the content of a class as the developer modifies the source code This technology is used in, for example, the products JRebel and XRebel from https://zeroturnaround.com/. Although Java agents work in the deep details of Java, they are not magic. They are a bit complex and you need a deep understanding of Java, but anyone who can program in Java can write a Java agent. All that is required is that the class, which is the agent, has some predefined methods packaged into a JAR file along with the other classes of the agent and has a META-INF/MANIFEST.MF file that defines the names of the classes implementing the premain and/or agentmain methods, and some other fields. The detailed and precise reference documentation is part of the JDK JavaDoc available at http://download.java .net/java/jdk9/docs/api/ in the documentation of the java.lang.instrument package. When a Java application is started with a Java agent, the command line has to contain the following option: -javaagent:jarpath[=options] Here, jarpath points to the JAR file that contains the agent class and the manifest file. The class must have a method named premain or agentmain. It may have one or two arguments. The JVM tries to call the twoargument version first after the JVM is initialized: public static void premain(String agentArgs, Instrumentation inst); If a two-argument version does not exist, then the one-argument version is used, which is essentially the same as the two-argument version but misses the instrumentation argument, which, in my opinion, does not make too much sense since a Java agent cannot do much without the Instrumentation object: public static void premain(String agentArgs); The agentArgs parameter is the string passed as an option on the command line. The second argument, Instrumentation, provides methods to register class transformers that can modify class byte codes and also methods that can ask the JVM to perform redefinition or retransformation of classes during runtime. Java applications can also load an agent after the program has already started. In such a case, the agent cannot be invoked before the main method of the Java application, since it has already started by that time. To separate the two cases, JVM calls agentmain in such a scenario. Note that either premain or agentmain is invoked for an agent and never both. A single agent can implement both so that it is capable of performing its task loaded at the startup, specified on the command line or after the JVM started. If agentmain is used, it has the same arguments as premain. There is one major and important difference between the invocation of premain and agentmain. If an agent cannot be loaded during startup, for example, if it cannot be found, if the JAR file does not exist, if the class does not have the premain method, or if it throws an exception, the JVM will abort. If the agent is loaded after the JVM is started (in this case, agentmain is to be used), the JVM will not abort if there is some error in the agent. This approach is fairly reasonable. Imagine that there is a server application that runs on the Tomcat servlet container. When a new version is started, the system is down for a maintenance period. If the new version cannot be started because the agent is not behaving well, then it is better not started. The damage to debug the situation and fix it, or roll back the application to the old version and call for a longer fixing session may be less than starting up the application and not having the proper agent functionality. If the application starts up only without the agent, then the suboptimal operation may not immediately be recognized. On the other hand, when an agent is attached later, the application is already running. An agent is attached to an already running application to get information from an already running instance. To stop the already running instance and fail it, especially in an operational environment, is more damaging than just not attaching the agent. It may not go unnoticed anyway because the agent that is most probably attached is used by operational personnel. A premain or agentmain agent gets an Instrumentation object as the second argument. This object implements several methods. One of them is: void addTransformer(ClassFileTransformer transformer) The agent implements the transformer, and it has the transform method signature: byte[] transform(Module module, ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException This method is called by the JVM when a class is loaded or when it is to be transformed. The method gets the class object itself, but, more importantly, it gets the byte array containing the byte code of the class. The method is expected to return the byte code of the transformed class. Modifying the byte code needs some knowledge of how the byte code is built and what the structure of a class file is. There are libraries that help to do that, such as Javassist (http://www.javassist.org/ ) or ASM (http://asm.ow2.org/). Nevertheless, I will not start coding before getting acquainted with the structure of the byte code. Agents, running in a separate thread and presumably interacting with the user or the filesystem and based upon some external observation at any time, may call the following method to perform the retransformation of the classes using the registered transformers: void retransformClasses(Class<?>... classes) The agent can also call the following method, which will redefine the classes given as arguments: void redefineClasses(ClassDefinition... definitions) The ClassDefinition class is simply a Class and a byte[] pair. This will redefine the classes through the class maintaining mechanism of the JVM. Note that these methods and Java agents interact with the deep, low-level part of the JVM. This also bears the consequence that it is very easy to destroy the whole JVM. The byte code is not checked, unlike during the loading of the class, and thus, if there is some error in it, the consequence may not only be an exception but also the crashing of the JVM. Also, the redefinition and the transformations should not alter the structure of the classes. They should not change their inheritance footprint, add, rename, or remove methods, or change the signature of the methods, and this is also true for fields. Also note that the already created objects will not be affected by the changes; they will still use the old definition of the class and only new instances will be affected. Polyglot programming Polyglot programming is the technique when there are different programming languages used in the same application. Such an approach is not only appropriate when a different part of the application runs on a different environment. For example, the client executes in the browser using JavaScript, CSS, and HTML while the server is programmed to run in a Tomcat environment in Java. This is a different story, and, usually, this is not the typical use when someone is speaking about polyglot programming. When the application that runs on the server partially runs in Java and also in some other language, then we can speak about polyglot programming. For example, we create the order handling application in Java and some of the code that checks the correctness of the order based on the product-specific codes that the order contains is written in JavaScript. Does it ring a bell? We have already done that in this book to demonstrate the scripting API of the JDK. That was real polyglot programing even if we did not mention it that way. The JVM that runs the compiled Java code is a very good target for different language compilers, and thus, there are many languages that compile for it. When the JVM runs the byte code of a class, it does not know what the source language was, and it does not really care; some compiler created the byte code and it just executes that. We can use different languages, such as Jython, Groovy, and Scala, to name a few popular ones that compile for the JVM. We can write one class using one language and the other one using another. When they are put together into a JAR, WAR, or an EAR file, the runtime system will just run them. When do we use polyglot programming? Polyglot configuration Usually, we turn towards polyglot programming when we want to create an application that is more flexible and more configurable. Applications that get installed in many instances, usually, at different customer sites have some configurations. These configurations can be XML files, properties files, and INI files (those come from Windows). As the programs develop sooner or later, these static configuration possibilities reach their limits. Application developers soon see that they need to configure some functionality that is cumbersome to describe using these technologies. Configuration files start being larger and, also, the code that reads and interprets the configuration files grow large. Good developers have to realize that this is the situation, and before the configuration files and the code handling them become unmanageable, some scripting configuration, polyglot programming has to be implemented. Decent developer teams may reach a point when they develop their configuration language and the interpreter of that language. It can be based on XML, or it can just be any other language. After all, writing a language is fun; I have done it a few times myself. Most of these were, however, hobbies and not professional projects. Usually, there is no customer value in crafting another language. We can better use an existing one. In the case of configuration, Groovy is a very handy language that supports complex closure and metaclass syntax and implementation. This way, the language is extremely suitable to create a domain-specific language. Since Groovy is compiled to JVM, Groovy classes can be invoked directly from Java, and in the other way round, reading the configuration is essentially invoking the class compiled from the configuration file. The compilation can be during application build time, but in the case of configuration, it makes more sense to do it during application startup. We have already seen that the Groovy implementation of the scripting API or the special API that Groovy provides is absolutely capable of doing that. Have we seen examples of this in our book? It may be a surprise to you, but we have in fact used Groovy to describe some configuration many times. Gradle build files are nothing more than Groovy DSL developed mainly in Groovy to support project build configuration. Polyglot scripting Configuration is not the only application of polyglot programming. Configuration is executed at the program startup and the configuration data is used as static data afterwards. We can execute scripts during the application's execution any time and not only during its startup. This can be used to provide extra functionality to the program's user with installations that use the same application but are furnished with different scripts. One of the first applications that provided such scripting capability was the emacs editor. The core of the application was written in C language and it contained a Lisp interpreter that let the user to write scripts, which were executed in the editor environment. The engineering program, AutoCAD, also used a Lisp interpreter for similar purposes. Why was Lisp used for this purpose? Lisp has very simple syntax, and therefore, it is easy to parse Lisp code. At the same time, the language is powerful, and last but not least, there were open source Lisp interpreters (at least one) available by the time. To get this kind of flexibility, applications, many times, provide plugin APIs, which a developer can use to extend the application. This, however, requires that the developer sets up coding tools, including IDE, build tool, continuous integration, and so on, that is, a professional programming environment. When the task to be solved by the plugin is simple, the overhead is simply too large. In such a case, a scripting solution is handier. Scripting is not a solution for everything. When the scripts extending the application tend to become too complex, it means that the scripting possibility is just too much. It is difficult, however, to take back a toy from a child. If users get used to the scripting possibility, then they will not take it easy if the next version of the application we release does not provide that possibility. Thus, it is extremely important to assess the possible use of the scripting capability in our application. Scripting and, more generally, any feature of our program will not be used for what we intended them for. They will be used for whatever it is possible to use them for. Users can go beyond all imagination when it comes to abusing some feature. It may be a good idea to think about limiting the scripting possibility beforehand, limiting the running time of the scripts or the size of the script our program agrees to work with. If these limitations are set reasonably, and the users understand and accept these, a plugin structure in addition to the scripting capability has to be considered. The security of an application, including plugin or scripting extension, is also very important. The scripts or plugins run on the same JVM as the core application. Some scripting languages provide some fence around the scripts that limits the access to the core application's objects and classes, but this is an exception. Usually, scripts run with the same privilege as the core application and that way they can do just anything. Thus, scripts should be trusted the same way as the core application. Script installation or modification should never be possible for an unprivileged user of the application. Such an action is almost always reserved for the system administrator. If an unprivileged user can upload a script to the server and then have it executed, we just opened a security hole in our application. Since access restrictions are enforced by the application, it is easy to override these limitations using an uncontrolled script. The hacker can just access other users' data easily, which he is not entitled to, and read and modify our database. Business DSL Polyglot programming may also come into the picture when the application's code can be separated into business code and technology code. The business code contains the top-level business logic that we actually write the application for, and this is the code that contains the logic that the customer pays for. The technology code is to support the algorithms coded in the business DSL. Most of the enterprise applications contain these two types of code but many do not separate them. This leads to a monolithic application that contains repetitive code. When you feel that you are writing the same type of code when you need persistence or networking, and again the same type of code while coding some business rules, then this is the code smell that suggests that the two code types are not separated. DSL and scripting are not a magic wand and do not solve all the problems that stem from a wrong application structure. In such a situation, the code has to be refactored first to separate the business logic and the infrastructure code, and it is only the second step to implement a DSL and a business API supporting it and to rewrite the business code into the DSL. Every step of such a project delivers value for the application and even if it never gets to DSL and scripting, the effort invested is not wasted. The business DSL scripting is very similar to pluggable scripts, except that this time it is not the application that calls the scripts from time to time to execute some special extension functionality. Instead, the DSL code calls the application through the business API that it provides. The advantage of providing the API and using a DSL is that the code that implements the business logic gets rid of the technical details, can be very abstract, and, this way, be much closer to a business-level description of the problem rather than just program code. Even some businessperson can understand a business DSL, and though it is not a goal in real-life examples, they could even write code. At TU Vienna, we also used a similar approach to make semiconductor simulation more usable for the semiconductor design engineer. The core calculating code was written in Fortran. A C language framework that handled the massive simulation data input and output and that embedded the XLISP interpreter executed these programs. The Lisp code contained the simulation configuration data and could also contain simple loops when the simulation was to be executed for many configuration points. It was polyglot programming, except that we did not know that this is going to be the name years after this application coding style. Problems with polyglot Polyglot programming is not only all about advantages. Before jumping into this direction, developers making the decision have to consider a lot of things. Using another language for the application needs knowledge. Finding people who can code in the languages that are used is eventually more difficult than finding developers who only know Java. (This is also true if the kernel application language is not Java.) Different languages require different mindsets and, many times, different people. The team should also have some members who are proficient in both languages, and it is also an advantage if most of the people know at least a bit about the other language. The toolset supporting Java is outstanding. The build tools, integrated development environment, libraries, debugging possibilities, and logging frameworks, to name a few, are all extremely good compared with other languages. Polyglot development needs support for the other language as well, which may not be as advanced as the support for Java. Many times, it is really an issue to debug DSL solutions and IDE support may also be lagging. When we program in Java, many times, we take for granted that the IDE reads the meta-data of the libraries and whenever we need to call a method, or reference a class, the IDE suggests the best possibility. XML and properties files may also be supported and the IDE may know some of the most used frameworks, such as Spring, and understand the XML configuration handling the names of the classes as hyperlinks, even when the class names are inside some attribute strings. This is far from being this easy in the case of other languages. For the languages that have a wide user base, the tooling support may be good, but if you pick some exotic language, you are on your own. The more exotic the language the less support you may have. You can create some tool to support your DSL that you develop. It is not hard to do so using tools such as http://www.eclipse.org/Xtext/. In such a case, you are tied to Eclipse, which may or may not be a problem. You can pick a special language, for example, Kotlin, which is extensively supported by IntelliJ, because the same company supports the language and the IDE, but again, you buy into a special technology that can be expensive to replace in case you have to. It is generally true not only for languages but also for any technology you include into your development. When you select one, you should consider the support and the cost of getting off the horse if or when it starts dying. Annotation processing We have already discussed annotations in great detail. You may recall that we defined our annotation interfaces using the following annotation: @Retention(RetentionPolicy.RUNTIME) This told the Java compiler to keep the annotation and put it into the JVM code so that the code can access it during runtime using reflection. The default value is RetentionPolicy.CLASS, which means that the annotation gets into the byte code, but the JVM does not make it available for the runtime system. If we use RetentionPolicy.SOURCE, the annotation does not even get into the class file. In this case, there is only one possibility to do anything with the annotation: compile time. How can we write code that runs during compile time? Java supports the notion of annotation processors. If there is a class on the classpath of the compiler that implements the javax.annotation.processing.Processor interface, then the compiler will invoke the implemented methods one or more times, passing information about the source file that the compiler is actually processing. The methods will be able to access the compiled methods, classes, or whatever is annotated, and also the annotation that triggered the processor invocation. It is important, however, that this access is not the same as in runtime. What the annotation processor accesses is neither a compiled nor a loaded class, that is, it is available when the code uses reflection. The source file at this time is under compilation; thus, the data structures that describe the code are actually structures of the compiler, as we will see in our next example. The annotation processor is called one or more times. The reason it is invoked many times is that the compiler makes it possible for the annotation processors to generate source code based on what it sees in the partially compiled source code. If the annotation processor generates any Java source file, the compiler has to compile the new source code and perhaps compile some of the already compiled files again. This new compilation phase needs annotation processor support until there are no more rounds to execute. Annotation processors are executed one after the other, and they work on the same set of source files. There is no way to specify the order of the annotation processor executions; thus, two processors working together should perform their tasks, no matter in what order they are invoked. Also, note that these codes run inside the compiler. If an annotation processor throws an exception, then the compilation process will most probably fail. Thus, throwing an exception out of the annotation processor should only be done if there is an error that cannot be recovered and the annotation processor decides that the compilation after that error cannot be complete. When the compiler gets to the phase to execute the annotation processors, it looks at the classes that implement the javax.annotation.processing.Processor interface and creates instances of these classes. These classes have to have a public no-argument constructor. To streamline the execution of the processors and to invoke a processor only for the annotations that it can handle, the interface contains two methods: to return the latest version the annotation processor can support getSupportedAnnotationTypes to return a set of String objects containing the fully qualified class name of getSupportedSourceVersion the annotations that this processor can handle If an annotation processor was created for Java 1.8, it may work with Java 9, but it may also not work. If it declares that the latest supported version is 1.8, then the compiler in a Java 9 environment will not invoke it. It is better not to invoke an annotation processor than calling it and messing up the compilation process, which may even create compiled but erroneous code. The values returned by these methods are fairly constant for an annotation processor. An annotation processor will return the same source version it can handle and will return the same set of annotations. Therefore, it would be clever to have some way to define these values in the source code in a declarative manner. It can be done when we extend the javax.annotation.processing.AbstractProcessor class instead of directly implementing the Processor interface. This abstract class implements these methods. Both of them get the information from the annotation so that we can decorate the class that extends the abstract class. For example, the getSupportedAnnotationTypes method looks at the SupportedAnnotationTypes annotation and returns an array of annotation type strings that are listed in the annotation. Now, this is a bit brain twisting and can also be confusing at first. We are executing our annotation processor during compile time. But the compiler itself is a Java application, and in this way, the time is runtime for the code that runs inside the compiler. The code of AbstractProcessor accesses the SupportedAnnotationTypes annotation as a runtime annotation using reflection methods. There is no magic in it. The method in the JDK 9 is as follows: public Set<String> getSupportedAnnotationTypes() { SupportedAnnotationTypes sat = this.getClass().getAnnotation (SupportedAnnotationTypes.class); if (sat == null) { ... error message is sent to compiler output ... return Collections.emptySet(); } else return arrayToSet(sat.value()); } (The code has been edited for brevity.) To have an example, we will sort of look at the code of a polyglot annotation processor. Our very simple annotation processor will process one simple annotation: com.javax0.scriapt.CompileScript, which can specify a script file. The annotation processor will load the script file and execute it using the scripting interface of Java 9. This code was developed as a demonstration code by the author of this book a few years ago and is available with the Apache license from GitHub. Thus, the package of the classes is retained. The annotation processor contains two code files. One of the annotation itself that the processor will work on: @Retention(RetentionPolicy.SOURCE) @Target(ElementType.TYPE) public @interface CompileScript { String value(); String engine() default ""; } As you can see, this annotation will not get into the class file after compilation; thus, there will be no trace during runtime so that any class source may occasionally use this annotation. Target of the annotation is ElementType.TYPE, meaning that this annotation can only be applied to those Java 9 language constructs that are some kind of types: class, interface, and enum. The annotation has two parameters. The value should specify the name of the script file, and the engine may optionally define the type of the script that is in that file. The implementation we'll create will try to identify the type of the script from the filename extension, but if somebody would like to bury some Groovy code into a file that has the .jy extension (which is usually for Jython), so be it. The processor extends AbstractProcessor and, in this way, some of the methods are inherited at the expense of some annotations used in the class: package com.javax0.scriapt; import ... @SupportedAnnotationTypes("com.javax0.scriapt.CompileScript") @SupportedSourceVersion(SourceVersion.RELEASE_9) public class Processor extends AbstractProcessor { There is no need to implement the getSupportedAnnotationTypes and getSupportedSourceVersion methods. These are replaced by the use of the annotations on the class. We support only one annotation in this processor, the one that we defined in the previously listed source file, and we are prepared to manage the source code up to Java version 9. The only method we have to override is process: @Override public boolean process( final Set<? extends TypeElement> annotations, final RoundEnvironment roundEnv) { for (final Element rootElement : roundEnv.getRootElements()) { try { processClass(rootElement); } catch (Exception e) { throw new RuntimeException(e); } } return false; } This method gets two arguments. The first is the set of annotations that it was invoked for. The second is the round environment. Because the processor can be invoked many times, the different invocations may have different environments. Each invocation is in a round and the RoundEnvironment argument is an object that can be used to get information about the given round. It can be used to get the root elements of the round for which this annotation is invoked. In our case, this will be a set of class elements that have the CompileScript annotation. We iterate over this set, and for each class, we invoke the processClass method (see the next code snippet). The method may throw some checked exception and the method process cannot because it should match the same method of the interface. Thus, we catch any exception that may be thrown and we re-throw these encapsulated in RunTimeException. If any of these exceptions are thrown by the called method, then the compilation could not run the scripts and it should be treated as failed. The compilation should not succeed in such a case: private void processClass(final Element element) throws ScriptException, FileNotFoundException { for (final AnnotationMirror annotationMirror : element.getAnnotationMirrors()) { processAnnotation(annotationMirror); } } The actual annotation is not available during compile time as we already mentioned. Hence, what we have available is only a compile time mirror image of the annotation. It has the AnnotationMirror type, which can be used to get the actual type of the annotation and, also, the values of the annotation. The type of the annotation is available during compile time. The compiler needs it; otherwise, it could not compile the annotation. The values are available from the annotation itself. Our processAnnotation method handles each annotation it gets as an argument: private void processAnnotation( final AnnotationMirror annotationMirror) throws ScriptException, FileNotFoundException { final String script = FromThe.annotation(annotationMirror). getStringValue(); final String engine = FromThe.annotation(annotationMirror). getStringValue("engine"); execute(script, engine); } Our @CompileScript annotation defines two parameters. The first value is the script filename and the second one is the scripting engine name. If the second one is not specified, then an empty string is set as the default value. The execute method is called for each and every occasion of the annotation: private void execute(final String scriptFileName, final String engineName) throws ScriptException, FileNotFoundException { final ScriptEngineManager factory = new ScriptEngineManager(); final ScriptEngine engine; if (engineName != null && engineName.length() > 0) { engine = factory.getEngineByName(engineName); } else { engine = factory.getEngineByExtension (getExtensionFrom(scriptFileName)); } Reader scriptFileReader = new FileReader (new File(scriptFileName)); engine.eval(scriptFileReader); } The method tries to load the script, based on the filename, and tries to instantiate the script engine, based on the given name. If there is no name given, then the filename extension is used to identify the scripting engine. By default, the JavaScript engine is on the classpath as it is part of the JDK. If any other JVMbased scripting engine is in use, then it has to be made available on the classpath or on the module path. The last method of the class is a simple script manipulation method, nothing special. It just chops off the filename extension so that the engine can be identified based on the extension string: private String getExtensionFrom(final String scriptFileName) { final int indexOfExtension = scriptFileName.lastIndexOf('.'); if (indexOfExtension == -1) { return ""; } else { return scriptFileName.substring(indexOfExtension + 1); } } And just for the sake of completeness, we have the closing brace of the class: } Programming in the enterprise When a professional works for an enterprise, she does not work alone. There are a lot of people, developers as well as other coworkers, we have to cooperate with. The older the IT department of the enterprise is, and the larger the enterprise is, the more specialized roles people are in. You will certainly meet business analysts, project managers, test engineers, build engineers, subject-matter experts, testers, architects, scrum masters, and automation engineers, to name a few roles. Some of these roles may overlap, no person may have more than one responsibility, and while in other cases, some roles could even be more specialized. Some of the roles are very technical and require less business-related knowledge; others are more business oriented. Working together as a team with so many people and with so many different roles is not simple. The complexity of the task may be overwhelming for a novice developer and cannot be done without definite policies that all members of the operation follow, more or less. Perhaps your experience will show that it is more times less than more, but that is a different story. For the way developers work together, there are well-established industry practices. These support the Software Development Lifecycle (SDLC) using waterfall, agile, or a mix of the two models in some way. In the following sections, we will look at tools and techniques that are, or at least should have been, used in every software development organization. These are: Static code analysis tools that control the quality of the code examining the source code Source code version control that stores all the versions of the source code and help get the source code for any old version of the development Software versioning to keep some order of how we identify the different versions and do not get lost among the different versions Code review and tools that help in pin-pointing bugs that are not revealed by tests and aid knowledge sharing Knowledge base tools to record and document the findings Issue tracking tools that record bugs, customer issues, and other tasks that somebody has to attend to Selection process and considerations for external products and libraries Continuous integration that keeps the software in a consistent state and reports immediately if there is some error in it before the error propagates to other versions or other code, depending on how the erroneous code gets developed Release management, which keeps track of the different release versions of the software Code repository, which stores the compiled and packed artifacts The following diagram shows the most widely used tools for these tasks: Static code analysis Static code analysis tools read the code just like the compiler and analyze it, but instead of compilation, they try to find errors or mistakes in it. Not the syntax errors. For that, we already have the Java compiler. Mistakes, such as using a loop variable outside a loop, which may be absolutely valid but is usually bad style and, many times, such usage comes from some simple mistakes. They also check that the code follows the styling rules that we set. Static code analyzers help identify many small and obvious errors in the code. Sometimes, they are annoying, warning about something that may not be really a problem. In such a case, it is better to code the program a bit differently, not because we want the static code analysis to run without warning. We should never modify the code because of a tool. If we code something in such a way that it passes some quality check tool and not because it is better that way, then we are serving the tools instead of the tools serving us. The reason to change the code to pass the code analysis is that it is very probable that the code is more readable to an average programmer if it does not violate the coding style. You or the other team members can be excellent programmers who understand the code very easily even if it uses some special construct. However, you cannot say that about all the programmers who will maintain your code in the future. The code lives a long life. I work with some programs that have been written 50 years ago. They are still running and maintained by young professionals around the age of 30. It means that they were not even born when the code was developed. It can easily happen that the person maintaining your code is not even born by the time you write the code. You cannot tell anything about their cleverness and coding practices. The best we can do is to prepare for the average and that is exactly what static code analysis tools are set for. The checks that these tools perform are not hardwired into the tools. Some special language inside the tools describes the rules and they can be deleted, other rules can be added, and rules can be modified. This way, you can accommodate the coding standards of the enterprise you work for. The different rules can be categorized as cosmetic, minor, major, and critical. Cosmetic things are mainly warnings and we do not really care about them, even though it is nice to fix even these issues. Sometimes, these small things may signal some really big issue. We can set limits for the number of minor and major bugs before the check is declared as failing and also for the critical errors. In the last case, this limit is usually zero. If a coding error seems to be critical, then better not have any in the code. The most frequently used tools are Checkstyle, FindBugs, and PMD. The execution of these tools is usually automated, and though they can be executed from the IDE or from the developer's command line, their main use is on the continuous integration (CI) server. During the build, these tools are configured on the CI server to run, and it can be configured such that the build should be broken if the static code analysis fails with some limit. Executing the static code analysis is usually the next step after compilation and unit test execution, and before the actual packaging. The SonarQube tool (https://www.sonarqube.org/) is a special tool in addition to being a static code analysis tool. SonarQube maintains the history of the previous checks as well as supports unit test code coverage and can report the change of the quality over time. This way, you can see how the quality, coverage percentage, and number of different qualifications of code style errors have changed. Many times, you can see that when approaching the release date, the code quality decreases as people are in a rush. This is very bad because this is the time when most of the bugs should be eliminated. Having a statistic about the quality may help change the practice by seeing the trends before the quality, and thus the maintainability of the code gets out of hand. Source code version control Source code version control systems store different versions of the source code. These days, we cannot imagine professional software development without it. This was not always the case, but the availability of free online repositories encouraged hobby developers to use some version control, and when these developers worked for enterprises later, it was evident that the use of these systems is kind of a must. There are many different revision control systems. The most widely used one is Git. The version control that was previously widely used was SVN and, even before that, CVS. These are less and less used these days. We can see SVN as a successor of CVS and Git as a successor of SVN. In addition to these, there are other version control systems such as Mercurial, Bazaar, or Visual Studio Team Services. For a comprehensive list of the available tools, visit the Wikipedia page at https://en.wikipedia.org/wiki/List_of_version_con trol_software. My bet is that you will meet Git in the first place and there is a high probability of you coming across SVN when programming for an enterprise. Mercury may appear in your practice but any of the others that currently exist are very rare, are used for a specific area, or are simply extinct. Version control systems allow the development team to store the different versions of the software in an organized manner on a storage that is maintained (backed up regularly in a reliable manner). This is important for different purposes. The first thing is that different versions of the software may be deployed to different instances. If we develop software for clients and we have many clients with whom we hope to have to make a terrific business, then different clients may have different versions. This is not only because some clients are reluctant to pay for the upgrade, and we just do not want to give the new version for free. Many times, the costs that rise on the side of the customer prevent the upgrade for a long time. Software products do not work on their own in an isolated environment. Different clients have different integrated environments; the software communicates with different other applications. When a new version is to be introduced in an enterprise environment, it has to be tested for whether it works with all the systems it has to cooperate with. This testing takes a lot of effort and money. If the new features or other values that the new version delivers over the old one do not justify the cost, then it would be waste of money to deploy the new version. The fact that there is a new version of our software does not mean that the old versions are not usable. If there is some bug at the customer's end, then it is vital that we fix the bug in that version. To do so, the bug has to be reproduced in the development environment, which eventually means that the source code for that version has to be available for the developers. This does require the customer database to contain references to the different versions of our software products that are installed at the customer site. To make it more complicated, a customer may have more than one version at a time in different systems and may also have different licenses, so the issue is more complex than it first seems. If we do not know which version the client has, then we are in trouble. Since the database registering the versions for the customers and real life may get unsynchronized, software products log their version at startup. We have a separate section about versioning in this chapter. If the bug is fixed in the version that the client has, the incident at the customer's end may be solved after deployment. The problem, though, still remains if the version is not the previous version of the software. The bug fix introduced to an old version of the software may still be lurking around in the later or, for that matter, earlier versions. The development team has to identify which versions are relevant to clients. For example, an old version that is not installed any more at any of the clients' sites does not deserve the investigation. After that, the relevant versions have to be investigated to check whether they exhibit the bug. This can only be done if we have the source version. Some old versions may not have the bug if the code causing the bug is introduced in later versions. Some new versions may also be immune to the bug because the bug was already fixed in the previous version, or simply because the piece of code that caused the bug was refactored even before the bug manifested. Some bugs may even affect a specific version instead of a range of products. Big fixing may be applied to different versions and they may need slightly different fixes. All this needs a maintained source version repository. Even when we do not have different customers with different versions, it is more than likely that we have more than one version of our software in development. The development of a major release is coming to an end, and therefore, one part of the team responsible for testing and bug fixing focuses on those activities. At the same time, the development of features for the next version still goes on. The code implementing the functionalities for the next version should not get into the version that is about to be released. The new code may be very fresh, untested, and may introduce new bugs. It is very common to introduce freeze times during the release process. For example, it may be forbidden to implement any new feature of the upcoming release. This is called feature freeze. Revision control systems deal with these freeze periods, maintaining different branches of the code. The release will be maintained in one branch and the version for later releases in a different one. When the release goes out, the bug fixes that were applied to it should also be propagated to the newer version; otherwise, it might so happen that the next version will contain bugs that were already fixed in the previous version. To do so, the release branch is merged with the ongoing one. Thus, version control systems maintain a graph of the versions, where each version of the code is a node in the graph and the changes are vertices. Git goes very far in this direction. It supports branch creation and merging so well that developers create separate branches for each change that they create and then they merge it back with the master branch when the feature development is done. This also makes for a good opportunity for code review. The developer making the feature development or bug fix creates a pull request in the GitHub application, and another developer is requested to review the change and perform the pull. This is a kind of four-eyed principle applied to code development. Some of the revision control systems keep the repository on a server and any change gets to the server. The advantage of this is that any change committed gets to a server disk that is regularly backed up and is thus safe. Since the server-side access is controlled, any code sent to the server cannot be rolled back without trace. All versions, even the wrong versions, are stored on the server. This may be required by some legal control. On the other hand, if commit requires network access and server interaction, it may be slow and this will, in the long run, motivate developers not to commit their changes frequently. The longer a change remains on the local machine, the more risk we have of losing some of the code, and merging becomes more and more difficult with time. To heal this situation, Git distributes the repository and the commit happens to the local repository, which is exactly the same as the remote one on some server. The repositories are synchronized when one repository pushes the changes to another one. This encourages the developers to make frequent commits to the repository, giving short commit messages, which helps in tracking the change made to the code. Some older version control systems support file locking. This way, when a developer checks out a code file, others cannot work on the same piece of code. This essentially avoids the collisions during code merging. Over the years, this approach did not seem to fit the development methodologies. Merge issues are less of a problem than files that are checked out and forgotten. SVN supports file locking but this is not really serious and does not prevent one developer to commit changes to a file that somebody else locked. It is more of only a suggestion than real locking. Source code repositories are very important but should not be confused with release repositories, which store the compiled released version of the code in binary. Source and release repositories work together. Software versioning Software versioning is magic. Think about the different versions of Windows or Star Wars movies. Well, the latter is not really software versioning but it shows that the issue is very general. In the case of Java, versioning is not that complex. First of all, the version of Java we use now is 9. The previous version was 1.8, before that 1.7, and so on, down to 1.0. Earlier versions of Java were called Oak but that is history. After all, that is, who can tell what Java 2 was? Fortunately, when we create a Java application, the situation is simpler. There has been a suggestion from Oracle, from the time of Java 1.3, about how to version JARs: http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/versioning.html This document distinguishes between specification version and implementation version. If the specification of a JAR content changes, the code has to behave differently from how it was behaving till then; the specification version should change. If the specification is not changed but the implementation does--for example, when we fix a bug--then the implementation version changes. In practice, nobody has used this scheme, although it is a brilliant idea to separate the implementation and specification versions, at least, in theory. I even bet that most of your colleagues have not even ever heard about this versioning. What we use in practice is semantic versioning. Semantic versioning (http://semver.org/) mixes the specification and implementation versions into one single version number triplet. This triplet has the format of mmp, that is: m: major version number m: minor version number p: patch number The specification says that these numbers start with zero and increase by one. If the major number is zero, it means that the software is still in development. In this state, the API is not stable and may change without a new major version number. The major version number gets to 1 when the software is released. Later it has to be increased when the API of the application (library) has changed from the previous version and the application is not backward compatible with the previous version. The minor version number is increased when the change effects only the implementation but the change is significant, perhaps, even the API is also changing but in a backward-compatible manner. The patch version is increased when some bug is fixed, but the change is not major and the API does not change. The minor and the patch levels have to be reset to zero if any version number in the triplet in front of any of them is increased: major version number increase resets both minor and patch version; minor version number increase resets patch number. This way, semantic versioning keeps the first element of the triplet for the specification version. The minor is a mix of the specification and implementation versions. A patch version change is clearly an implementation version change. In addition to these, semantic versioning allows appending a pre-release string, such as -RC1 and -RC2. It also allows the appending of metadata, such as a date after a plus sign, for example, +20160120 as a date. The use of semantic versioning helps those that use the software to easily spot compatible versions and to see which version is older and which is newer. Code review When we create programs in a professional way, it is done in teams. There is no one-man show programming other than in a hobby or going along with the tutorials. It is not only because it is more effective to work in teams but also because one person is vulnerable. If you work alone and get hit by the bus or you hit the lottery and lose your ability or motivation to work on the project, your customer is in trouble. That is not professional. Professional projects should be resilient to any member falling off. Teamwork needs cooperation and one form of cooperation is code review. This is the process when a developer or a group of developers reads a part of the code that some other team members have written. There are direct gains from this activity; The developers reading the code get more knowledge about the code; they learn the code. This way, if the developer creating the code gets out of the process for any reason, the others can continue the work with minimal bump. Coding styles can be aligned. Developers, even seniors, paying careful attention make coding mistakes. This may be a bug or it may be a coding style violation. Coding style is important because the more readable the code is, the less possibility of it having unnoticed bugs. (Also see the next bullet point.) It is also important that the coding style is the same for the team. All team members should use the same style. Looking at a code that has a different style from the one I wrote is a bit harder to follow and understand. The differences may distract the reader and the team members have to be able to read the code. The code belongs to the team and not a single developer. Any team member should know the code and be able to modify it. During code review, a lot of bugs can be discovered. The parties looking at the code and trying to understand the working of it may occasionally discover bugs from the structure of the code, which are otherwise hard to discover using tests. If you want, code review is the whitest white box test. People think differently and different mindsets catch different bugs. Code review can be done online and offline. It can be done in teams or peer-to-peer. Most teams follow the code review process that GitHub supports, which is the simplest. Changes to the code are committed to a branch and are not merged with the code directly but, rather, a pull request is created on the web interface. The local policy may require that a different developer perform the pull. The web interface will highlight the changes and we can add comments to the changed code. If the comments are significant, then the original developer requesting the pull should modify the code to answer the comments and request the pull again. This ensures that at least two developers see any change; the knowledge is shared. Feedback is peer-to-peer. It is not a senior teaching a junior. That needs a different channel. Comments in GitHub are not good for this purpose; at least, there are better channels. Perhaps talking face to face. Comments may come from a senior to a junior or from a junior to a senior. In this work, giving feedback on the quality of the code, seniors and juniors, are equal. The simplest and perhaps the most frequent comment is the following: I can see that Xyz.java was changed in the modification but I see no change made to XyzTest.java. This is almost an instant refusal for the merge. If a new feature is developed, unit tests have to be created to test that feature. If a bug is fixed, then unit tests have to be created to prevent the bug from coming back. I personally got this comment many times, even from juniors. One of them told me, "We know that you were testing us if we dared to give feedback." God knows I was not. They did not believe. While change review and GitHub is a good tool during development, it may not be appropriate when a larger chunk of code has to be reviewed. In such a case, other tools, such as FishEye, have to be used. In this tool, we can select the source files for review even if they were not recently changed. We can also select reviewers and deadlines. Commenting is similar to GitHub. Finally, this type of code review finishes with a code review session, where the developers gather and discuss the code in person. While organizing such a session, it is important that a person who has experience managing other people mediates these sessions. Code and discussion on styles can get very personal. At the same time, when attending such a meeting, you should also pay attention so as not to get personal. There will be enough participants who may not know this or are less disciplined. Never attend a review session without reviewing the code first using the online tools. When you make comments, the language should be very polite for the reason I have already mentioned. Finally, the mediator of the meeting should be able to separate important and not so important issues and to stop debate on bagatelle things. Somehow, the less important issues are more sensitive. I personally do not care about formatting the tab size if it is two or four spaces and if the file should contain only spaces or if tab characters are allowed, but people tend to like to waste time on such issues. The most important issue during code review sessions is that we are professional and it may happen that I review and comment your code today, but tomorrow, it will be just the opposite, and we work together and we have to work together as a team. Knowledge base Knowledge base was a buzzword a few years ago. Few companies were evangelizing the idea of wiki technology and nobody was using it. Today, the landscape of knowledge base is totally different. All enterprises use some kind of wiki implementation that is there to share knowledge. They mostly use Confluence, but there are also other wiki solutions available, commercial and free as well. Knowledge bases store information that you, as a developer, would write down in a paper notebook for your later reference, for example, the IP address of the development server, directories where to install the JAR files, what commands to use, what libraries you have collected, and why you use them. The major difference is that you write it in a formatted way into a wiki and it is available immediately for other developers. It is a bit of a burden on the developer to write these pages, and it needs some selfdiscipline first. Sticking to the example of the IP address of the development server and the install directories, you have to write not only the IP address of the server but also some text explaining what the information is, because the others may not understand it otherwise. It is also a bit of work to place the page with the information in the wiki system with a good name, linking it to other pages, or finding the appropriate position of the page in the tree of pages. If you were using the paper notebook, you could just write down the IP address and the directories on the first free page of the book and you would just remember all others. The wiki approach will pay back when coworkers do not need to find the information themselves; you can find the information in an easier way because other coworkers have also recorded their findings in the knowledge base and, last but not least, a few months later, you find the information you recorded yourself. In the case of a paper notebook, you would turn the pages to find the IP address and you may or may not remember which one is the primary and which is the secondary server. You may even forget by then that there are two servers (or was it a double cluster?). To have a long list of available wiki software, visit https://en.wikipedia.org/wiki/Comparison_of_wiki_software. Issue tracking Issue tracking systems keep track of issues, bugs, and other tasks. The first issue tracking systems were created to maintain the list of bugs and also the state of the bug fixing process to ensure that a bug, identified and recorded, will not get forgotten. Later, these software solutions developed and became fullfledged issue trackers and are unavoidable project management tools in every enterprise. The most widely used issue tracking application used in many enterprises is Jira, but on the https://en.wikipedia .org/wiki/Comparison_of_issue-tracking_systems page, you can find many other applications listed. The most important feature of an issue tracker application is that it has to record an issue in detail in an editable manner. It has to record the person who recorded the issue in case more information is needed during issue handling. The source of the issue is important. Similarly, issues have to be assigned to some responsible person, who is accountable for the progress of issue handling. Modern issue tracking systems provide complex access control, workflow management, relation management, and integration with other systems. Access control will only allow the person who has something to do with an issue access to it, so others cannot alter the state of an issue or even read the information attached to it. An issue may go through different workflow steps depending on the type of issue: a bug may be reported or reproduced, a root cause analyzed, a fix developed or tested, a patch created, a fix merged with the next release version or published in the release. This is a simple workflow with a few states. Relation management allows setting different relations between issues and allowing the user to navigate from issue to issue along these relations. For example, a client reports a bug, and the bug is identified as being the same as another already fixed. In such a case, it would be insane to go through the original workflow and creating a new patch for the same bug. Instead, the issue gets a relation pointing to the original issue and sets the state to be closed. Integration with other systems is also useful to keep a consistent development state. Version control may require that, for every commit, the commit message contains a reference to the issue that describes the requirement, bug, or change that the code modification supports. Issues may be linked to knowledge base articles or agile project management software tools using web links. Testing We have already discussed testing when we talked about unit testing. Unit testing is extremely important in agile development and it helps keep the code clean and reduce the number of errors. But this is not the only type of testing that you will see in enterprise development. Types of tests Testing is performed for many reasons but there are at least two reasons that we have to mention. One is to hunt the bugs and create error-free code as much as possible. The other is to prove that the application is usable and can be utilized for the purpose it was meant for. It is important from the enterprise point of view and considers a lot of aspects that unit test does not. While unit test focuses on one unit and, thus, is an extremely good tool to point out where the error is, it is totally unusable when it comes to discovering bugs that come from erroneous interfaces between modules. The unit tests mock external modules and, thus, test that the unit works as expected. However, if there is an error in this expectation and the other modules do not behave in the same way as the unit test mock, the error will not be discovered. To discover the errors on this level, which is the next level above unit test, we have to use integration tests. During integration tests, we test how individual units can work together. When we program in Java, the units are usually classes; thus, the integration test will test how the different classes work together. While there is a consensus (more or less) about what a unit test is in Java programming, this is less so in the case of integration tests. In this regard, the external dependencies, such as other modules reachable via the network or database layers may be mocked, or may be set up using some test instance during integration testing. The argument is not about whether these parts should be mocked or not, only the terminology. Mocking some components such as the database has advantages as well as drawbacks. As in the case of any mock, the drawback is the cost of setting up the mock as well as the fact that the mock behaves differently from the real system. Such a difference may result in some bugs still remaining in the system and lurking there until a later case of testing or, God forgive, production is used. Integration tests are usually automated in a way similar to unit tests. However, they usually require more time to execute. This is the reason why these tests are not executed at each source code change. Usually, a separate maven or Gradle project is created that has a dependency on the application JAR and contains only integration test code. This project is usually compiled and executed daily. It may happen that daily execution is not frequent enough to discover the integration issues in a timely manner, but a more frequent execution of the integration tests is still not feasible. In such a case, a subset of the integration test cases is executed more frequently, for example, every hour. This type of testing is called smoke testing. The following diagram shows the position of the different testing types: When the application is tested in a fully set up environment, the testing is called system testing. Such testing should discover all the integration bugs that may have been lurking and covered during the previous testing phases. The different type of system tests can also discover non-functional issues. Both functional testing and performance testing are done on this level. Functional testing checks the functions of the application. It ensures that the application functions as expected or at least has functions that are worth installing in the production environment and can lead to cost saving or profit increase. In real life, programs almost never deliver all the functions that were envisioned in any requirement documentation, but if the program is usable in a sane manner, it is worth installing it, assuming that there are no security issues or other issues. In case there are a lot of functions in the application, functional testing may cost a lot. In such a case, some companies perform a sanity test. This test does not check the full functionality of the application, only a subset to ensure that the application reaches a minimal quality requirement and it is worth spending the money on the functional testing. There may be some test cases that are not envisioned when the application was designed and thus there is no test case in the functional test plan. It may be some weird user action, a user pressing a button on the screen when nobody thought it was possible. Users, even if benevolent, happen to press or touch anything and enter all possible unrealistic inputs into a system. Ad-hoc testing tries to amend this shortage. A tester during ad-hoc testing tries all the possible ways of use of the application that he or she can imagine at the moment the test is executed. This is also related to security testing, also called penetration testing when the vulnerabilities of the system are discovered. These are special types of tests that are performed by professionals who have their core area of expertise in security. Developers usually do not have that expertise, but at least, the developers should be able to discuss issues that are discovered during such a test and amend the program to fix the security holes. This is extremely important in the case of Internet applications. Performance testing checks that the application, in a reasonable environment, can handle the expected load that the user puts on the system. A load test emulates the users who attack the system and measures the response times. If the response time is appropriate, that is, lower than the required maximum under the maximum load, then the test passes; otherwise, it fails. If a load test fails, it is not necessarily a software error. It may so happen that the application needs more or faster hardware. Load tests usually test the functionality of the application in only a limited way and only test use scenarios that pose read load on the application. Many years ago, we were testing a web application that had to have a response time of 2 seconds. The load test was very simple: issue GET requests so that there are a maximum of 10,000 requests active at the same time. We started with 10 clients, and then a script was increasing the concurrent users to 100, then 1,000, and then stepping up by thousand every minute. This way, the load test was 12 minutes long. The script printed the average response time, and we were ready to execute the load test at 4:40 pm on a Friday. The average response time started from a few milliseconds and went up to 1.9 seconds as the load was increased to 5,000 concurrent users, and from there, it was descending down to 1 second as the load was increased to 10,000 users. You can understand the attitude of the people on a Friday afternoon, being happy that we met the requirements. My colleagues left for the weekend happily. I remained testing a bit more because I was bothered by the phenomenon that the response time decreases as we increase the load above 5,000. First, I reproduced the measurement and then started looking at the log files. At 7 pm, I already knew what the reason was. When the load went above 5,000, the connections the Apache server was managing started to exhaust and the web server started to send back 500 internal error codes. That is something that Apache can very effectively do. It is very fast in telling you that you cannot be served. When the load was around 10,000 concurrent users, 70% of the responses already had 500 errors. The average went down, but the users were actually not served. I reconfigured the Apache server so that it could serve all the requests and forward each to our application just to learn that the response time of our application was around 10 seconds at the maximum load. Around 10 pm, when my wife was calling my mobile the third time, I also knew how large a memory I should set in the Tomcat startup file in the options for the JVM to get the desired 2-second response time in case of 10,000 concurrent users. Stress test is also a type of performance test that you may also face. This type of test increases the load on the system until it cannot handle the load. That test should ensure that the system can recover from the extreme load automatically or manually but, in no case, will do something that it shouldn't at all. For example, a baking system should not ever commit an unconfirmed transaction, no matter how big the load there is. If the load is too high, then it should leave the dough raw but should not bake extra bread. The most important test at the top of the hierarchy is the user acceptance test. This is usually an official test that the customer, who buys the software, executes and in the case of successful execution, pays the price for the software. Thus, this is extremely important in professional development. Test automation Tests can be automated. It is not a question of whether it is possible to automatize a test, only whether it is worth doing so. Unit tests and integration tests are automated, and as time advances, more and more tests get automated as we move along to higher steps towards the user acceptance test (UAT). UAT is not likely to be automated. After all, this test checks the integration between the application and the user. While the user, as an external module, can be mocked using automation in lower levels, we should reach the level when the integration test happens without mocks. There are many tools that help the automation of tests. The blocker for test automation, these days, is the cost of the tools to do so, the cost of learning and developing the tests, and the fear that the automated tests are not discovering some of the errors. It is true that it is easier to do something wrong with a program than without. This is so true for almost anything not only for testing. And still we do use programs; why else would you read this book? Some of the errors may not be discovered during automated functional testing, which would otherwise have been discovered using manual tests. At the same time, when the same test is executed the hundredth time by the same developer, it is extremely easy to skip an error. An automated test will not ever do that. And most importantly, the cost of the automated test is not 100 times the cost of running it once. We have used test automation tools in this book. SoapUI is a tool that helps you create tests that can be executed automatically. Other testing tools that are worth looking at are Cucumber, Concordion, Fintnesse, and JBehave. There is a good comparison of tools at https://www.qatestingtools.com/. Black box versus white box You may have heard many times that a test is a black box test. This simply means that the test does not know anything about how the system under test (SUT) is implemented. The test relies only on the interface of the SUT that is exported for the outside world. A white box test, on the other end of the scale, tests the internal working of the SUT and very much relies on the implementation: Both the approaches have advantages and disadvantages. We should use one, or the mixture of the two approaches, a way that fits the purpose of the actual testing needs the most. A black box test not relying on the implementation does not need to change if the implementation changes. If the interface of the tested system changes, then the test should also be changed. A white box test may need changes if the implementation changes, even if the interface remains the same. The advantage of the white box test is that, many times, it is easier to create such a test and the testing can be more effective. To get the best of both worlds, systems are designed to be testable. Be careful, though. It means many times that the functionality internal to the tested system is propagated to the interface. That way, the test will use only the interface and, thus, can be declared to be a black box, but it does not help. If something changes in the internal working of the tested system, the test has to follow it. The only difference is that you may call it a black box test if the interface also changes. That does not save any work. Rather, it increases it: we have to check all the modules that rely on the interface if they also need any change. I do not say that we should not pay attention to creating testable systems. Many times making a system testable results in cleaner and simpler code. If the code, however, gets messier and much longer because we want to make it testable, then we are probably not going in the right way. Selecting libraries Programming for the enterprise or even programming a moderately sized project cannot be done without the use of external libraries. In the Java world, most of the libraries that we use are open source and, more or less, free to use. When we buy a library that is sold for money, there is usually a standard process enforced by the purchasing department. In such a case, there is a written policy about how to select the vendor and the library. In the case of "free" software, they do not usually care, though they should. In such a case, the selection process mainly lies with the IT department and it is therefore important to know the major points to be considered before selecting a library even if for free. In the previous paragraph, I put the word free between quotes. That is because there is no software, which is free. There is no such thing as a free lunch, as they say. You have heard this many times but it may not be obvious in the case of an open source code library or framework you are going to select. The major selection factor for any purchase or implementation is the cost, the price. If the software is free, it means that you do not need to pay an upfront fee for the software. However, there is a cost in integrating it and using it. Support costs money. Somebody may say that the support is community support and also available free of charge. The thing is that the time you spend hunting for a workaround that helps you to get over a bug is still money. It is your time, or in case you are a manager, it is the time of the professional in your department whose time you pay for, or, as a matter of fact, it can be an external contractor who will hand you a huge bill in case you do not have the expertise in-house to solve the issue. Since free software does not have a price tag attached, we have to look at the other factors that are important in the selection process. At the end of the day, they all will affect the cost in some way. Sometimes, the way a criterion alters the cost is not obvious or easily calculable. However, for each one, we can set no-go levels that are based on technology decisions, and we can compare libraries for being better or worse along with each of the criteria. Fit for the purpose Perhaps, this is the most important factor. Other factors may be argued about in terms of the scale of importance, but if a library is not appropriate for the purpose we want to use, then this is certainly not something to select, no matter what. It may be obvious in many cases, but you may be surprised how many times I have seen a product selected because it was the favorite of a person in some other project and the library was forced for use in the new project even though the requirements were totally different. License The license is an important question as not all free software is free for all uses. Some of the licenses allow free use for hobby projects and education but require you to purchase the software for professional, profit-oriented use. The most widely used licenses and their explanation (and the whole text of the license) is available on the web page of the Open Source Initiative (https://opensource.org/licenses). It lists nine different licenses, and to make the situation a bit more complex, these licenses have versions. One of the oldest licenses is the General Public License (GPL) standing for GNU. This license contains the following sentences: For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. If you create software for a for-profit enterprise and the company intends to sell software, you probably cannot use any line of code that is from a GPL-licensed software. It would imply that you are required to pass on your own source code, which may not be the best sales strategy. Apache license, on the other hand, may be okay for your company. This is something that the lawyers should decide. Even though this is the lawyers' work, there is one important point that we developers must be aware of and pay close attention to. Sometimes, the libraries contain code from other projects and their license, as advertised, may not be the real one. A library may be distributed under the Apache license but contains code that is GPL-licensed. This is obviously a violation of the GPL license, which was committed by some open source developers. Why would you care? Here comes the explanation via an imagined situation. You develop software for an enterprise. Let's say that this company is one of the largest car manufacturers of the world, or it is one of the largest banks, pharma, whatever. The owner of the GPL software seeks remedies for the misuse of her software. Will she sue the software developer, John Doe, who has a total wealth of 200K, or your company, claiming that you did not duly check the license of the code? She certainly will not dig for gold where there is none. Suing the company you work for may not be successful, but certainly not a good process you or anyone at the company wants. What can we as software professionals do? We have to use libraries that are well known, used widely. We can check the source code of the library to see whether there is some copied code. Some package names may present some clue. You can Google some part of the source code to find matches. Last but not least, the company can subscribe to services that provide similar research for the libraries. Documentation Documentation is an important aspect. If the documentation is not appropriate, it will be hard to learn how to use the library. Some of the team members may have already known the library, but, again, this may not be the case for later team members. We should consider our colleagues, who are expected to be average programmers, and they will have to learn the use of the library. Thus documentation is important. When we speak about documentation, we should not only think about the JavaDoc reference documentation but also tutorials and books if they are available. Project alive It is important not to select a library for use that is not alive. Have a look at the roadmap of the library, the last time a release was shipped, and the frequency of the commits. If the library is not alive, we should consider not using it. Libraries work in an environment and the environment changes. The library may connect to a database. The new version of the database may provide new features that give us better performance only if the library is modified to accommodate these new features. The library communicates over HTTP; will it support the new 2.0 version of the protocol? If nothing else, the version of the Java environment will change over the years and the library we use should sooner or later follow it to leverage the new features. There is no guarantee that an alive library will always stay alive. However, a library that is already dead will certainly not resurrect. Even if the project is alive at the moment, there are some points that may give some hints about the future of the library. If the company developing it is well-established and financially stable, and the library is developed with a reasonable business model, then there is a low risk that the project dies. If there are a lot of companies who use the library, then it is likely that the project will stay alive even if the original team stops working on it or the original financing structure changes. However, these are only small factors and not well-established facts. There is no guarantee, and telling the future is more an art than a science. Maturity Maturity is similar to the previous criterion. A project may very well be alive just starting up, but if it is in its infancy, we better not use the library for a large project. When a project is in its early phase, a lot of bugs can be in the code, the API may change radically, and presumably, there may only be a small number of companies relying on the code. This also means that the community support is lower. Of course, if all the projects select only mature open source code, then no open source project would ever get to the mature state. We should assess the importance of the project. Is the project business-critical? Will the project become business-critical? If the project is not business-critical, the company may afford to invent a fresh library that is not that mature. It may be reasonable if there are no mature libraries for the purpose because the technology you are going to use is relatively new. In such a case, the project in the company is probably also new and not business-critical yet. It will be business-critical, we hope, after some time, but by that time, the library will be mature, or may just die and we can select a competing solution before the project becomes too expensive to switch. Judging the maturity of a library is always difficult and has to be aligned with the maturity and importance of the project that we want to use the library for. Number of users If the library is alive and mature but there are not many users, then something is smelly. Why don't people use the library if it is good? If the number of users for a library or framework is low and there are no large corporations among the users, then it is probably not a good one. Nobody using it may signal that our assessment of the other criteria may not be appropriate. Also note that if there are only a few users of the library, then the knowledge in the community is also scarce and we may not be able to get community support. The "I like it" factor Last but not least, the I like it factor is extremely important. The question is not whether you like the library but rather how much the developers like it. Developers will like a library that is easy to use and fun to work with, and this will result in low cost. If the library is hard to use and developers do not like it, then they will not learn to use it to the level of profession required for good quality, only to the level that is just needed. The end result will be suboptimal software. Continuous integration and deployment Continuous integration means that whenever a new version is pushed to the source code repository, the continuous integration server kicks in, pulls the code to its disk, and starts the build. It compiles the code first, then runs the unit tests, fires the static code analysis tools, and, if all goes right, it packages a snapshot release and deploys it on a development server. CI servers have web interfaces that can be used to create a release. In such a case, the deployment can even go to the test servers or even to production depending on local business needs and on the policy that was created accordingly. Automating the build and deployment process has the same advantages as any other automation: repeated tasks can be performed without manual intervention, which is tedious, boring, and, thus, error-prone if done by a human. The outstanding advantage is that if there is some error in the source code that can be discovered by the automated build process, it will be discovered. Novice developers say that it is cheaper and easier to build the code locally, which the developers do anyway, and then push the code to the server if the build process is already checked. It is partly true. Developers have to check that the code is of good quality and builds well, before sending it to the central repo. However, this cannot always be achieved. Some errors may not manifest on local environments. It may so happen that one developer accidentally uses a newer version of Java than the one supported and uses a new feature of the new version. Enterprises do not generally use the latest technology. They tend to use versions that are proven, have many users, and are mature. This year, in 2017, when Java 9 is going to be released in July, huge enterprises still use Java 1.6 and 1.7. Since Java 9 has many new features that are not trivial to implement, I expect that the adoption of the technology may take even longer than the adoption of Java 1.8, which gave us functional programming and lambda. It may also happen that a new library is added to the dependencies of the build and the developer who added it to the build file (pom.xml, or build.gradle) could use it without any problem on her local machine. It does not mean that the library is officially added to the project, and it may not be available in the central code repository (Artifactory, Nexus, or other implementations of the code repository). The library may have only been on the local repository of the developer, and she may have assumed that since the code compiles, the build is OK. Some large organizations use different code repositories for different projects. The libraries get into these repositories following meticulous examination and decisions. Some libraries may get there, while others may not. The reason to have different repositories could be numerous. Some project is developed for one customer who has a different policy about an open source project than the other. If the enterprise develops code for itself, it may so happen that some library is phased out or not supported anymore, and can only be used for projects that are old. A maintenance release may not need to replace a library, but new projects may be not be allowed to use a dying software library. The CI server can run on a single machine or it can run on several machines. In case it serves many projects, it may be set up as a central server with many agents running on different machines. When some build process has to be started, the central server delegates this task to one of the agents. The agents may have different loads, running several different build processes, and may have different hardware configuration. The build process may have requirements regarding the speed of the processor or about the available memory. Some agent may run simpler builds for smaller projects but would fail to execute the build of a large project or of some small project that still has a huge memory requirement to execute some tests. When a build fails, the build server sends out e-mails to the developers, and the person who sent the last update to the code repository is obligated to fix the bug without delay. This encourages the developers to commit frequently. The smaller the change, the fewer chances there are of a build problem. The build server web interface can be used to see the actual state of the projects, which project is failing to build, and which is just fine. If a build fails, there is a red sign in the line of the build, and if the build is OK, the sign is green. Many times, these reports are continually displayed on some old machine using a huge display so that every developer or just anybody who enters the room can see the actual state of the builds. There is even special hardware that you can buy that has red, yellow, and green lamps to follow the state of the build and ring a bell when the build fails. Release management Developing software means a continuously changing code base. Not every version of the software is supposed to be installed in production. Most of the versions are pushed to the repository on a branch half complete. Some versions are meant only for testing and a few are meant to be installed in production even if only some of those will finally get to production. Almost all the time, the releases follow the semantic versioning that we discussed in an earlier section. The versions that are meant only to be tested usually have the -SNAPSHOT modifier at the end of the version number. For example, the 1.3.12-SNAPSHOT version is the version that was once debugged, and is going to become the 1.3.12 version. The snapshot versions are not definite versions. They are the code as it is by then. Because a snapshot release never gets installed in production, it is not needed to reproduce a snapshot version for maintenance. Thus, the snapshot versions are not increased continually. Sometimes, they may be changed, but that is a rare exception. It may so happen that we work on a bug fix, 1.3.12-SNAPSHOT, and during the development, we change so much code that we decide that it has to be 1.4.0 when it is released, and we rename the snapshot as 1.4.0SNAPSHOT. This is a rare case. Many times, the release creation creates a 1.4.0 version from 1.3.12-SNAPSHOT as the decision about the new release number is taken by the time the release is created. When the release process is started, usually from the web interface of the CI server, the developer creating the release has to specify the release version. This is usually the same as the snapshot version without the -SNAPSHOT postfix. The build process not only creates the build in this case but also tags the source code repository version it was using and loads the packaged program (artifact) to the code repository. The tag can be used later to access the exact version of the source code that was used to create the release. If there is a bug in a specific version, then this version has to be checked out on a developer machine to reproduce the bug and find the root cause. If the build of a release fails, it can be rolled back, or you better just skip that release number and note it as a failed release build. An existing release can never have two versions. The source code is the only one that is for that release and the generated code has to be exactly the one in any storage. Subsequent compilation of the same source may result in slightly different code, for example, if a different version of Java is used to create the latter one. Even in such a case, the one that was created by the build server in the first place is the version that belongs to the release. When a bug is reproduced and the code is recompiled from the exact same source, it is already a snapshot version. Multiple releases may be possible from the same source version, for example, compiled with Java versions from 1.5 to 1.8 and version 9 but a single release always belongs to the exact same source code. If a release that was supposed to be a release version fails during QA checks, then a new release has to be created and the failed release has to be noted as such. The version that marketing uses to name the different versions should not have a relation to the technical version numbers we work with. Many times, it is, and it causes much headache. If you realize that the two are totally different things and one does not have to do anything with the other, life gets simpler. Look at the different versioning of the Windows operating system or Java. As marketing, Java used 1.0 then 1.1, but Java 1.2 was advertised as Java 2 and still the code contained 1.2 (which now seven major releases later also becomes 9 instead of 1.9) The last part of release management is that deployments should register the version numbers. The company has to know which release is installed on which server, and of which client. Code repository Code repository stores the libraries and helps manage the dependencies of the different libraries. In the old times, when Java projects used ANT as a build tool and without the later added Ivy dependency management, the libraries that were needed by a project were downloaded to the source code, usually to the lib library. If a library needed another library, then those were also downloaded and stored manually, and this continued until all the libraries that one of the already downloaded libraries needed were copied to the source code tree. This was a lot of manual work and, also, the library code was stored in the source code repository in many copies. A compiled library is not source code and has nothing to do in the source code repository. Manual work that can be automated has to be automated. Not because developers are lazy (yes, we are and we have to be) but because manual work is error prone and, thus, expensive. This was when Apache Ivy was invented and Maven, following ANT, already supported repository management built in. They all stored the libraries structured in directories and supported metadata that described the dependencies to other libraries. Lucky that Gradle did not invent its own code repository. Instead, it supports both Maven and Ivy repositories. Using the repository, the build tools automatically download the libraries that are needed. In case a library has a new version, then the developer only has to update the version of the needed library in the build configuration and all tasks, including downloading all the new versions of the other libraries that are needed by that version, are done automatically. Walking up the ladder At this point, you have got a lot of information that will rocket your start as an enterprise Java developer. You have got a base knowledge that you can build on. There is a long way to become a professional Java developer. There is a lot of documentation to read, a lot of code to scan and understand, and also a lot of code to write till you can claim to be a professional Java developer. You may probably face many years of continuous education. The good thing is that even after that, you can continue your journey and you can educate yourself, as being a professional Java developer is rarely a job people retire from. No, no! Not because they die while at it! Rather, professional software developers gaining experience start to code less and less and support the development process in different ways, which leverages more of their experience. They can become business analysts, project managers, test engineers, subject-matter experts, architects, scrum masters, automation engineers, and so on. Is it a familiar list? Yes, these are the people you will work with as a developer. Many of them may have started as a developer themselves. The following diagram shows the relative position of these roles: Let's take a bit more detailed look into what these roles perform in enterprise development: Business analysts work with the client and create the documents, specifications, use cases, and user stories needed by the developers to develop the code. Project managers administer the projects and help the team in getting things done in cooperation with other teams, caring for all the project matters that developers cannot attend to or would unnecessarily burn their time that they should have devoted to coding. Subject-matter experts are more advanced in knowing the business needs, so it is a bit rare for a developer to become one, but in case the industry you work in is technology oriented, it may not be incredible to become one. Test engineers control the QA process and understand not only the test methodologies and requirements of testing but also the development process so that they can support bug fixes and not only identify them, which would be poor. Architects work with BAs and design a high-level structure of the applications and code, and document it in a way that helps the developers to focus on the actual tasks they have to perform. Architects are also responsible for the solution to use technologies, solutions, and structures which fit the purpose, are future proof, affordable, and so on. Scrum mates help the development team to follow the agile methodology and help the team in controlling the administration and resolving problems. There are many ways to go as a software developer and I only listed some of the positions that you can find in an enterprise today. As technology develops, I can imagine that in 20 years from today, software developers will teach and curate artificial intelligence systems and that will be what we refer to as programming today. Who can tell? Summary Going in this direction is a good choice. Being a Java developer and becoming a professional in it is a profession that will pay well in the coming 10 to 20 years for sure and perhaps even later. At the same time, I personally find this technology fascinating and interesting, and after more than 10 years of Java programming and more than 35 years of programming, I still learn something new in it every day. In this book, you learned the basics of Java programming. From time to time, I also mentioned issues, suggested directions, and warned you about pitfalls that are not Java-specific. However, we also did the homework of learning the Java language, the infrastructure, the libraries, development tools, and networking in Java. You also learned the most modern approaches that came only with Java 8 and 9, such as functional programming in Java, streams, and reactive programming. If you know all that I have written in this book, you can start working as a Java developer. What's next? Go, and find your treasure in programming and in Java!