how to round a jbutton in java?

You can create a class that implements the Border interface.

import java.awt.*;
import javax.swing.*;
import javax.swing.JFrame;
import javax.swing.border.Border;

public class Roundbutton extends JFrame {

    private static final long serialVersionUID = 34534511L;

    public static void main(String[] args) {
        //       
        JFrame.setDefaultLookAndFeelDecorated(true);
        JFrame frame = new JFrame("Rounded Button Example");
        frame.setLayout(new FlowLayout());
        frame.setSize(150, 150);

        BordeRedondo border = new BordeRedondo(10); // radio = 10

        JButton addBtn = new JButton("X");
        addBtn.setBounds(10,10, 30, 30);
        addBtn.setBorder(border); // radio

        frame.add(addBtn);

        frame.setVisible(true);    

    }

}

class BordeRedondo implements Border {

    private int radio;  

    BordeRedondo(int radius) {
        this.radio = radius;
    }  

    public Insets getBorderInsets(Component c) {
        return new Insets(this.radio+1, this.radio+1, this.radio+2, this.radio);
    }  

    public boolean isBorderOpaque() {
        return true;
    }  

    public void paintBorder(Component c, Graphics g, int x, int y, int width, int height) {
        g.drawRoundRect(x, y, width-1, height-1, radio, radio);
    }

  }

In the new versions of Swing the tricks that were used in previous versions as null edge no longer work

setBorder(null)

or the empty edge trick

setBorder(BorderFactory.createEmptyBorder(0,0,0,0));

This is due to the routines that the default appearance best known as Metal Look And Feel have changed

So the option is to use another Look And Feel , I recommend using the which is called Nimbus, and which has been designed as a successor to Metal.

You must load it before showing your frame or you will not notice changes, you can do it like in this example when you start your main.

import java.awt.*;
import javax.swing.*;
import javax.swing.JFrame;
import javax.swing.UIManager.LookAndFeelInfo;

public class Roundbutton extends JFrame {

    private static final long serialVersionUID = 34534511L;

    public static void main(String[] args) {
        try {
            for (LookAndFeelInfo info : UIManager.getInstalledLookAndFeels()) {
                if ("Nimbus".equals(info.getName())) {
                    UIManager.setLookAndFeel(info.getClassName());
                    break;
                }
            }
        } catch (Exception e) {
            // If Nimbus is not available, you can set the GUI to another look and feel.
        }
        //       
        JFrame.setDefaultLookAndFeelDecorated(true);
        JFrame frame = new JFrame("Rounded Button Example");
        frame.setLayout(new FlowLayout());
        frame.setSize(150, 150);


        JButton addBtn = new JButton("X");

        frame.add(addBtn);

        frame.setVisible(true);

    }

}

The example looks like this: