Click the foreground or background color box of the Coloring tool on the Tools palette or right-click an object on the front panel or block diagram background with the Coloring tool to display the color picker. Click the button in the bottom right corner of the color picker to display this dialog box.

Use this dialog box to create a custom color.